Perimeter Monitoring System

Abstract

To provide a technique capable of more easily monitoring the peripheral situation of a work machine by using a plurality of periphery monitoring sensors.SOLUTION: A periphery monitoring system 100 comprises: a plurality of periphery monitoring sensors 120 mounted on a work machine (for example, a shovel SVL, a continuous unloader ULD, etc.) and acquiring information related to the peripheral situation of the work machine; and a position / posture detecting sensor 130 for acquiring information related to a position and a posture of each of the plurality of periphery monitoring sensors 120.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a perimeter monitoring system. To Regarding. [Background technology]

[0002] For example, a technique is known in which a plurality of peripheral monitoring sensors provided on a work machine are used to monitor the surrounding conditions (see Patent Documents 1 and 2).

[0003] In Patent Document 1, a camera is provided at each of the rear end, left end, and right end of the top surface of an upper rotating body of a shovel, and image information showing the situation around the shovel based on the output of the cameras is displayed on a display device inside the cabin.

[0004] In Patent Document 2, in a continuous unloader, images from both a camera mounted on the top of a bucket elevator that takes images of the opening of the hold, and a camera mounted at the tip of the bucket elevator that takes images of the inside of the hold, are displayed on a display in the control room. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2014-181510 A [Patent Document 2] JP 2016-160034 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, for example, in order to appropriately monitor the conditions around a work machine using multiple perimeter monitoring sensors, the positional relationship between the sensing areas of the multiple perimeter monitoring sensors and the work machine needs to be clear with a specified level of accuracy, which makes it necessary to perform calibration in advance to align the positional relationship between the sensing area of ​​each of the multiple perimeter monitoring sensors and the work machine to a specified relationship.

[0007] In view of the above-mentioned problems, an object of the present invention is to provide a technique that enables easier monitoring of the conditions around a work machine using a plurality of perimeter monitoring sensors. [Means for solving the problem]

[0008] In order to achieve the above object, in one embodiment of the present disclosure, A plurality of surrounding monitoring sensors mounted on the work machine for acquiring information regarding a surrounding situation of the work machine; a position and orientation detection sensor for acquiring information regarding the position and orientation of each of the plurality of perimeter monitoring sensors; a control device that integrates data relating to a situation around the work machine, the data being acquired by each of the plurality of periphery monitoring sensors, based on data acquired by the position and attitude detection sensor; Equipped with The plurality of periphery monitoring sensors are configured such that at least one of their relative positions and attitudes changes with the operation of the work machine. , 2 or more Includes a combination of perimeter monitoring sensors fruit , the control device integrates data relating to the surrounding conditions of the work machine, which is acquired by each of the two or more perimeter monitoring sensors, based on data relating to the positions and orientations of the two or more perimeter monitoring sensors, which is acquired by the position and orientation detection sensor; A perimeter monitoring system is provided. Effect of the Invention

[0010] According to the above-described embodiment, the situation around the work machine can be more easily monitored using a plurality of perimeter monitoring sensors. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a perimeter monitoring system. [Diagram 2] 1 is a side view showing an example of a shovel to which a perimeter monitoring system is applied. [Diagram 3] FIG. 1 is a top view showing an example of a shovel to which a perimeter monitoring system is applied. [Figure 4] FIG. 4 is a diagram showing an example of the display content of a display device. [Diagram 5] FIG. 11 is a diagram showing another example of the display content of the display device. [Figure 6] 1 is a diagram showing an example of a continuous unloader to which a perimeter monitoring system is applied; [Figure 7] 1 is a diagram showing an example of a continuous unloader to which a perimeter monitoring system is applied; [Figure 8] FIG. 1 is a diagram showing an example of a continuous unloader to which a perimeter monitoring system is applied. [Figure 9] FIG. 1 is a diagram showing an example of a continuous unloader to which a perimeter monitoring system is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment will be described with reference to the drawings.

[0013] [Outline of Perimeter Monitoring System] First, with reference to FIG. 1, an overview of a perimeter monitoring system 100 according to this embodiment will be described.

[0014] <Configuration> FIG. 1 is a schematic diagram showing an example of a perimeter monitoring system 100 according to the present embodiment.

[0015] The perimeter monitoring system 100 uses the output of the perimeter monitoring sensor 120 to monitor the situation around the work machine (for example, a shovel SVL or continuous unloader ULD, which will be described later) on which the perimeter monitoring sensor 120 is mounted. Hereinafter, unless otherwise specified, the "work machine" refers to the work machine whose surrounding situation is being monitored by the perimeter monitoring system 100.

[0016] The periphery monitoring system 100 includes a control device 110 , a periphery monitoring sensor 120 , a position and orientation detection sensor 130 , a display device 140 , and an input device 150 .

[0017] The control device 110 performs control related to monitoring of the conditions around the work machine. The control device 110 also provides information about the conditions around the work machine to users of the periphery monitoring system 100. Users of the periphery monitoring system 100 include, for example, operators of the work machine. The operators of the work machine may include operators who operate the work machine from outside the work machine, as well as operators who operate the work machine from a driver's cab provided on the work machine. Users of the periphery monitoring system 100 also include managers who monitor the operation of the work machine from outside (for example, a work machine management center, etc.).

[0018] The functions of the control device 110 may be realized by any hardware or any combination of hardware and software. The control device 110 is mainly configured with a computer including, for example, a central processing unit (CPU), a memory device such as a random access memory (RAM), a non-volatile auxiliary storage device such as a read only memory (ROM), and various interface devices for input and output. The control device 110 realizes various functions by, for example, loading a program installed in the auxiliary storage device into the memory device and executing the program on the CPU.

[0019] The control device 110 may be mounted on the work machine, or may be provided outside the work machine. When provided outside the work machine, the control device 110 may be, for example, a server device. The server device may be, for example, a cloud server provided in a management center or the like that is installed in a location different from the work site of the work machine. The server device may also be, for example, an edge server provided within the premises of the work site of the work machine, or in a communication facility (for example, a base station or station building) relatively close to the premises of the work site. When provided outside the work machine, the control device 110 may be a terminal device (for example, a desktop computer terminal) provided in a specified facility within the premises of the work site. When provided outside the work machine, the control device 110 may be, for example, a portable terminal device (mobile terminal) such as a smartphone, tablet terminal, or laptop computer terminal.

[0020] Furthermore, when the control device 110 is provided outside the work machine, it may be, for example, an external device (hereinafter, "remote operation support device") that supports remote operation of the work machine. The remote operation support device may be, for example, a server device or the like provided in a facility where an operator remotely operates a work machine. The remote operation support device is connected to an operation device (hereinafter, "remote operation device") through which the operator performs operation input corresponding to remote operation, a display device (hereinafter, "display device for remote operation") through which the operator visually checks image information around the work machine, and the like. In this way, the remote operation support device can obtain the contents of the operation input of the operator who remotely operates the work machine, transmit it to the work machine through a specified communication line, and reflect the contents of the remote operation in the operation of the work machine. Furthermore, the remote operation support device can display image information around the work machine received from the work machine through a specified communication line on the display device for remote operation, thereby supporting the remote operation of the user.

[0021] The periphery monitoring sensor 120 is mounted on the work machine and acquires data on the situation around the work machine. The periphery monitoring sensor 120 may include, for example, an imaging device that acquires image data of the periphery of the work machine. The imaging device may include, for example, a monocular camera, a stereo camera, a depth camera, a distance image camera, etc. The periphery monitoring sensor 120 may also include, for example, a distance sensor such as a LIDAR, an ultrasonic sensor, or a millimeter wave radar that acquires data on reflected signals from objects around the work machine. The distance sensor acquires data on the situation around the work machine by, for example, transmitting signals such as light, ultrasonic waves, and electromagnetic waves in a predetermined angular range up, down, left, and right from the sensor itself and receiving reflected signals that are reflected by objects in each direction from which the signals are transmitted and return. Specifically, the distance sensor acquires data on the received reflected signals (hereinafter, "received data"). The received data includes data on the time (TOF: Time Of Flight) from the emission of a signal to the reception of a reflected signal for each direction of the signal emission (hereinafter, "TOF data"). The received data also includes information on the strength of the signal received for each direction of signal irradiation (hereinafter, "received intensity data"). The distance sensor may output, for example, TOF (Time Of Flight) data, received intensity data, etc., as data on the conditions around the work machine. The distance sensor may also output point cloud data based on itself and attribute data of each point as data on the conditions around the work machine by applying known post-processing to the TOF data and received light intensity data, etc. The attribute data may be, for example, data indicating the type of object corresponding to each point.

[0022] The perimeter monitoring sensor 120 includes multiple perimeter monitoring sensors 120A, 120B, etc. Hereinafter, the perimeter monitoring sensors 120A, 120B, etc. may be collectively or individually referred to as "perimeter monitoring sensor 120X." The multiple perimeter monitoring sensors 120X may all be the same type of sensor, or may include two or more types of sensors.

[0023] The periphery monitoring sensor 120X outputs data relating to the situation around the work machine, and the output data of the periphery monitoring sensor 120X is taken into the control device 110.

[0024] When the control device 110 is mounted on a work machine, the output data of the periphery monitoring sensor 120X is input to the control device 110, for example, via a one-to-one communication sensor, an on-board network, or the like.

[0025] Furthermore, when the control device 110 is provided outside the work machine, output data from the periphery monitoring sensor 120X is input to the control device 110 through a predetermined communication line outside the work machine. The predetermined communication line may include, for example, a local network (LAN) inside the work site of the work machine. The predetermined communication line may also include, for example, a wide area network (WAN). The wide area network may include, for example, a mobile communication network terminated in a base station, a satellite communication network using a communication satellite, the Internet network, and the like. The predetermined communication line may also include, for example, a short-distance communication line according to a predetermined wireless communication standard such as Bluetooth (registered trademark) or WiFi.

[0026] The position and orientation detection sensor 130 acquires data on the position and orientation of each of the multiple perimeter monitoring sensors 120X that can be expressed in a predetermined coordinate system (hereinafter, "reference coordinate system"). The reference coordinate system may be, for example, a local coordinate system based on the work machine, or may be, for example, a global coordinate system such as the World Geodetic System. The reference coordinate system may be a fixed coordinate system or a moving coordinate system that moves together with a predetermined part of the work machine. The position and orientation detection sensor 130 may directly acquire data on the position and orientation of the perimeter monitoring sensor 120X expressed in the reference coordinate system, or may acquire data on the position and orientation of the perimeter monitoring sensor 120X in a form that can be converted into the reference coordinate system by a predetermined coordinate conversion.

[0027] The position and orientation detection sensor 130 may be one or more as long as it can output data related to the positions and orientations of all of the multiple perimeter monitoring sensors 120X. Furthermore, the position and orientation detection sensor 130 may be mounted on the work machine or provided outside the work machine as long as it can output data related to the positions and orientations of the multiple perimeter monitoring sensors 120X on the reference coordinate system.

[0028] The position and orientation detection sensor 130 includes, for example, an imaging device that outputs image data related to the position and orientation of the perimeter monitoring sensor 120X. In this case, for example, a predetermined figure (marker) or the like is provided at a predetermined portion of the perimeter monitoring sensor 120X that can be imaged by the position and orientation detection sensor 130. This allows the control device 110 to identify (estimate) the position and orientation of the perimeter monitoring sensor 120X based on the size, distortion, etc. of the figure on the image data output by the imaging device.

[0029] The position and orientation detection sensor 130 includes, for example, an RFID (Radio Frequency Identification) reader. In this case, the perimeter monitoring sensor 120X is equipped with, for example, a plurality of RFID tags installed at different locations. This allows the control device 110 to identify (estimate) the position and orientation of the perimeter monitoring sensor 120X based on the reception signals (received radio waves) from each of the plurality of RFID tags received by the RFID reader.

[0030] The position and orientation detection sensor 130 also includes a beacon receiver that receives, for example, electromagnetic waves or ultrasonic waves transmitted from a beacon according to a short-range communication standard such as WiFi or Bluetooth (registered trademark). In this case, the perimeter monitoring sensor 120X is equipped with, for example, multiple beacons installed in different locations. This allows the control device 110 to identify (estimate) the position and orientation of the perimeter monitoring sensor 120X based on the reception signals from each of the multiple beacons received by the beacon receiver.

[0031] The display device 140 displays image information showing the situation around the work machine under the control of the control device 110, and provides visual information to the user of the perimeter monitoring system 100. The display device 140 may be, for example, a stationary display device that is mounted on the work machine or installed in an external device (facility) that supports remote operation of the work machine. The display device 140 may also be a portable (transportable) display device that can be carried by the user of the perimeter monitoring system 100. Examples of portable display devices include wearable display devices, which will be described later. Examples of portable display devices include display devices (displays) mounted on mobile terminals, such as smartphones and tablet terminals.

[0032] The input device 150 accepts various inputs from a user of the perimeter monitoring system 100. The input device includes, for example, an operation input device such as a keyboard, a mouse, a joystick, a touch panel, a touch pad, a button, a toggle, a lever, etc. The input device may also include, for example, a voice input device or a gesture input device that accepts voice input or gesture input from the user. The input device 150 may be provided on the work machine or outside the work machine depending on the location of the target user.

[0033] <Operations related to monitoring the conditions around the work machine> The control device 110 integrates the output data of the multiple periphery monitoring sensors 120X into a data group that represents the surrounding conditions of the work machine in the reference coordinate system, based on the output data of the position and orientation detection sensor 130. Hereinafter, the integrated data group that represents the surrounding conditions of the work machine in the reference coordinate system will be referred to as an "integrated data group."

[0034] The control device 110 constructs an integrated data group based on the output data of the multiple periphery monitoring sensors 120X and the output data of the position and orientation detection sensor 130 inputted at each predetermined control cycle. The integrated data group may be constructed, for example, by linking the output data of each of the multiple periphery monitoring sensors 120X to a reference coordinate system. Specifically, the output data of the periphery monitoring sensor 120X corresponds to data in a coordinate system based on the periphery monitoring sensor 120X. Therefore, the integrated data group may be constructed by linking the output data of the periphery monitoring sensor 120X to data related to the position and orientation of the periphery monitoring sensor 120X in the reference coordinate system. The integrated data group may also be constructed, for example, by converting the output data of the periphery monitoring sensor 120X into data in the reference coordinate system.

[0035] For example, even if the position and orientation of the perimeter monitoring sensor 120X are fixed, there is a manufacturing error between the design position and the actual fixed position. Therefore, it is usually necessary to perform calibration (correction) on the position and orientation of the perimeter monitoring sensor 120X for all the perimeter monitoring sensors 120X before shipping from the factory.

[0036] Also, for example, the attitude of the perimeter monitoring sensor 120X may be changed by a user operation. This is because the location around the work machine that the user wants to grasp may vary depending on the work situation, etc. Also, the position and attitude of the perimeter monitoring sensor 120X may change in accordance with the operation of the moving parts of the work machine. In these cases, even if the calibration of the perimeter monitoring sensor 120X is performed in a specific position and attitude state, if at least one of the position and attitude is changed, it may become impossible to determine which area around the work machine the output data of the perimeter monitoring sensor 120X represents.

[0037] In contrast to this, in this embodiment, the position and orientation detection sensor 130 outputs data regarding the positions and orientations of the multiple perimeter monitoring sensors 120X in a form that can be expressed as positions and orientations in a reference coordinate system. Therefore, the control device 110 can grasp the positions and orientations of the multiple perimeter monitoring sensors 120X in the reference coordinate system regardless of the presence or absence of calibration, and can integrate the output data of the multiple perimeter monitoring sensors 120X as data in the same reference coordinate system. Therefore, the perimeter monitoring system 100 can more easily monitor the situation around the work machine.

[0038] Note that calibration (adjustment) regarding the position and orientation may be performed for the perimeter monitoring sensor 120X whose positional relationship with the position and orientation detection sensor 130 does not change. In this case, the position and orientation detection sensor 130 may output only data regarding the positions and orientations of the remaining perimeter monitoring sensors 120X, excluding the perimeter monitoring sensors 120X that have already been calibrated among the multiple perimeter monitoring sensors 120X.

[0039] Furthermore, when the position and attitude of the perimeter monitoring sensor 120X change in accordance with the operation of the movable parts of the work machine, a combination of two or more perimeter monitoring sensors 120X may occur in which at least one of the relative positions and attitudes changes in accordance with the operation of the work machine. In some cases, data of overlapping ranges around the work machine may be acquired by two or more perimeter monitoring sensors 120X, or the overlapping ranges may change. In this case, if the accurate positions and attitudes of the two or more perimeter monitoring sensors 120X are not grasped, data relating to overlapping ranges around the work machine (i.e., the same range) may be erroneously determined to be data relating to different ranges. As a result, the situation around the work machine may be erroneously recognized.

[0040] In contrast, in this embodiment, the control device 110 can appropriately recognize the positions and orientations of the multiple perimeter monitoring sensors 120X based on the output data of the position and orientation detection sensor 130. Therefore, even if a state occurs in which two or more perimeter monitoring sensors 120X acquire data of overlapping ranges around the work machine, or if the overlapping range of data changes, the control device 110 can appropriately recognize the data of the overlapping range as data of the same range. Therefore, even in such a situation, the perimeter monitoring system 100 can more appropriately recognize the situation around the work machine.

[0041] The control device 110 may, for example, recognize (detect) the position of an object in the vicinity of the work machine based on the integrated data set. Also, the control device 110 may, for example, recognize (detect) the type of the object based on the integrated data set.

[0042] For example, the control device 110 may recognize objects and their types around the work machine based on image data included in the integrated data group using a known image processing method or a machine learning-based classifier (e.g., a support vector machine), etc. Then, the control device 110 may recognize the position of the recognized object based on the position, size, etc. of the object on the image data.

[0043] Also, for example, the control device 110 may recognize (detect) objects around the work machine and recognize the positions of those objects based on the output data (point cloud data) of the distance sensor included in the integrated data group and data obtained by converting the point cloud data into a reference coordinate system. Furthermore, for example, the control device 110 may recognize the type of object around the work machine that has been recognized (detected) based on the output data (reception intensity data and attribute data) of the distance sensor included in the integrated data group. This is because the reflectance of the signal output from the distance sensor may differ depending on the type of object.

[0044] Furthermore, for example, when the integrated data group includes image data and output data of a distance sensor (or data after conversion thereof), the control device 110 may recognize the position of an object from the output data of the distance sensor and the type of the object from the image data. This is because the detection accuracy of the object position is relatively higher when the output data of the distance sensor is used, whereas the recognition accuracy of the type of the object is relatively higher when the image data is used.

[0045] Furthermore, the control device 110 may generate image information showing the situation around the work machine based on the integrated data group, for example, and display it on the display device 140 (see Figs. 4 and 5 described below). This allows the control device 110 to more easily generate a composite image with higher positional accuracy, for example, by using multiple data included in the integrated data group, and provide it to the user. This is because all of the data in the integrated data group is integrated in a unified reference coordinate system.

[0046] [Example of application of perimeter monitoring system to an excavator] Next, an example of application of the perimeter monitoring system 100 to an excavator SVL will be described with reference to Figs.

[0047] <Excavator configuration> 2 and 3 are a side view and a top view of an example of an excavator SVL to which the perimeter monitoring system 100 is applied.

[0048] <<Overview>> 2 and 3, an excavator SVL (an example of a work machine) according to this embodiment includes a lower traveling body 1, an upper rotating body 3, an attachment including a boom 4, an arm 5, and a bucket 6, and a cabin 10. Hereinafter, the front of the excavator SVL corresponds to a direction in which an attachment for the upper rotating body 3 extends when the excavator SVL is seen in a plan view from directly above along the rotation axis of the upper rotating body 3 (hereinafter simply referred to as a "plan view"). In addition, the left and right of the excavator SVL correspond to the left and right, respectively, as seen by an operator inside the cabin 10.

[0049] The lower traveling body 1 includes, for example, a pair of left and right crawlers 1C (specifically, a left crawler 1CL and a right crawler 1CR). The lower traveling body 1 causes the excavator SVL to travel by hydraulically driving each of the crawlers 1CL, 1CR by a traveling hydraulic motor 1M (specifically, a left traveling hydraulic motor 1ML and a right traveling hydraulic motor 1MR).

[0050] The upper rotating body 3 is mounted on the lower traveling body 1 so as to be rotatable via a turning mechanism 2. The upper rotating body 3 turns relative to the lower traveling body 1 as the turning mechanism 2 is hydraulically driven by a turning hydraulic motor 2A.

[0051] A boom 4 is attached to the front center of the upper rotating body 3 so as to be able to tilt up and down, an arm 5 is attached to the tip of the boom 4 so as to be able to rotate up and down, and a bucket 6 is attached to the tip of the arm 5 so as to be able to rotate up and down.

[0052] The bucket 6 is an example of an end attachment. The bucket 6 is used, for example, for excavation work. In addition, instead of the bucket 6, another end attachment may be attached to the tip of the arm 5 depending on the work content, etc. The other end attachment may be, for example, another type of bucket, such as a slope bucket or a dredging bucket. The other end attachment may be a type of end attachment other than a bucket, such as an agitator or a breaker.

[0053] The boom 4, the arm 5, and the bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, which serve as hydraulic actuators, respectively.

[0054] The cabin 10 is a cockpit where an operator sits, and is mounted, for example, on the front left side of the upper rotating body 3.

[0055] The excavator SVL operates actuators (e.g., hydraulic actuators) in response to the operation of an operator in the cabin 10, to drive driven elements such as the lower traveling body 1, upper rotating body 3, boom 4, arm 5, and bucket 6.

[0056] Further, instead of or in addition to being configured to be operable by the operator in the cabin 10, the shovel SVL may be configured to be remotely operated from outside the shovel SVL. When the shovel SVL is remotely operated, the inside of the cabin 10 may be unmanned. In the following description, it is assumed that the operation of the operator includes at least one of the operation of the operating device 26 by the operator in the cabin 10 and the remote operation by an external operator.

[0057] The remote operation includes, for example, a mode in which the excavator SVL is operated by an input from a user (operator) regarding the actuator of the excavator SVL performed by a predetermined external device (remote operation support device). In this case, the excavator SVL transmits image information (hereinafter, "peripheral image") of the surroundings of the excavator SVL based on the output of an imaging device (peripheral monitoring sensors 120A to 120C) described later to the remote operation support device. The peripheral image may be displayed on a remote operation display device provided in the remote operation support device. Also, various information images (information screens) displayed on the display device 140 in the cabin 10 of the excavator SVL may be similarly displayed on the remote operation display device of the remote operation support device. This allows the operator of the remote operation support device to remotely operate the excavator SVL while checking the display contents such as the peripheral image showing the state of the surroundings of the excavator SVL and various information images displayed on the remote operation display device. Furthermore, the excavator SVL may operate actuators and drive driven elements such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6 in response to a remote control signal representing the content of the remote control received from the remote operation support device.

[0058] Further, the remote operation may include a mode in which the shovel SVL is operated by, for example, an external voice input or gesture input to the shovel SVL by a person (e.g., a worker) around the shovel SVL. Specifically, the shovel SVL recognizes voices uttered by surrounding workers or gestures made by the workers through an imaging device or a voice input device (e.g., a microphone) mounted on the shovel SVL (own machine). Then, the shovel SVL may operate actuators according to the contents of the recognized voices, gestures, etc., to drive driven elements such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6.

[0059] Furthermore, the excavator SVL may automatically operate the actuators regardless of the contents of the operation by the operator, thereby realizing a function (so-called "automatic driving function" or "machine control function") of automatically operating at least some of the driven elements such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6.

[0060] The automatic driving function may include a function to automatically operate a driven element (actuator) other than the driven element (actuator) to be operated in response to an operator's operation of the operating device 26 or remote operation (so-called "semi-automatic driving function" or "operation-assisted machine control function"). The automatic driving function may also include a function to automatically operate at least a part of a plurality of driven elements (hydraulic actuators) on the premise that there is no operation or remote operation of the operating device 26 by the operator (so-called "fully automatic driving function" or "fully automatic machine control function"). In the excavator SVL, when the fully automatic driving function is enabled, the inside of the cabin 10 may be unmanned. The semi-automatic driving function, the fully automatic driving function, etc. may also include a mode in which the operation content of the driven element (actuator) to be the object of automatic driving is automatically determined according to a rule that is specified in advance. The semi-automatic driving function, the fully automatic driving function, etc. may also include a mode in which the excavator SVL autonomously makes various judgments and autonomously determines the operation content of the driven element (hydraulic actuator) to be the object of automatic driving in accordance with the judgment result (so-called "autonomous driving function").

[0061] The excavator SVL includes various components, such as a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside world, and a control system for various controls.

[0062] <<Hydraulic drive system>> 2 and 3, the hydraulic drive system of the excavator SVL includes hydraulic actuators that hydraulically drive each of the driven elements, such as the lower traveling body 1 (crawlers 1CL, 1CR), upper rotating body 3, boom 4, arm 5, and bucket 6, as described above. The hydraulic actuators include traveling hydraulic motors 1ML, 1MR, swing hydraulic motor 2A, boom cylinder 7, arm cylinder 8, and bucket cylinder 9. The hydraulic drive system of the excavator SVL according to this embodiment also includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0063] The engine 11 is the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, at the rear of the upper rotating body 3. The engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by a control device 110 (described later), and drives the main pump 14 and the pilot pump 15.

[0064] The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the control device 110. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter, the "tilt angle") in response to a control command from the control device 110.

[0065] The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the control device 110, the tilt angle of the swash plate is adjusted by the regulator 13 to adjust the stroke length of the piston, thereby controlling the discharge flow rate (discharge pressure).

[0066] The control valve 17 controls the hydraulic actuators in response to the contents of the operator's operation of the operating device 26 or remote operation, or operation commands related to the automatic operation function output from the control device 110. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and selectively supplies hydraulic oil supplied from the main pump 14 to the hydraulic actuators (travel hydraulic motors 1ML, 1MR, swing hydraulic motor 2A, boom cylinder 7, arm cylinder 8, bucket cylinder 9, etc.) in response to the operating state of the operating device 26 or operation commands output from the control device 110.

[0067] <<Operation system>> As shown in FIGS. 2 and 3, the operating system of the excavator SVL according to this embodiment includes a pilot pump 15 and an operating device 26.

[0068] The pilot pump 15 supplies pilot pressure to various hydraulic devices via a pilot line 25. The pilot pump 15 is mounted on the rear of the upper rotating body 3, for example, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

[0069] The operation device 26 is provided near the cockpit of the cabin 10, and is used by the operator to operate various driven elements (lower traveling body 1, upper rotating body 3, boom 4, arm 5, bucket 6, etc.). In other words, the operation device 26 is used by the operator to operate hydraulic actuators (i.e., traveling hydraulic motors 1ML, 1MR, swing hydraulic motor 2A, boom cylinder 7, arm cylinder 8, bucket cylinder 9, etc.) that drive the respective driven elements. The operation device 26 includes, for example, lever devices that operate each of the boom 4 (boom cylinder 7), arm 5 (arm cylinder 8), bucket 6 (bucket cylinder 9), and upper rotating body 3 (swing hydraulic motor 2A). The operation device 26 also includes, for example, pedal devices or lever devices that operate each of the left and right crawlers (traveling hydraulic motors 1ML, 1MR) of the lower traveling body 1.

[0070] For example, the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 uses hydraulic oil supplied from the pilot pump 15 through a pilot line to output a pilot pressure corresponding to the operation content to the control valve 17. As a result, pilot pressure corresponding to the operation content related to various driven elements (hydraulic actuators) in the operating device 26 can be input to the control valve 17. Therefore, the control valve 17 can drive each hydraulic actuator according to the operation content of the operating device 26 by an operator or the like.

[0071] Also, for example, the operating device 26 is of an electrical type. Specifically, the operating device 26 outputs an electric signal (hereinafter, "operation signal") corresponding to the operation content, and the operation signal is taken in by the control device 110. Then, the control device 110 outputs a control signal corresponding to the content of the operation signal to a hydraulic control valve (hereinafter, "operation hydraulic control valve") that can output hydraulic oil to the control valve 17 using hydraulic oil from the pilot pump 15. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the operation hydraulic control valve to the control valve 17, and the control valve 17 can drive each hydraulic actuator according to the operation content of the operating device 26.

[0072] <<User Interface>> As shown in FIGS. 2 and 3, the user interface system of the excavator SVL according to this embodiment includes the operation device 26, a display device 140, and an input device 150.

[0073] Display device 140, under the control of controller 110, provides visual information.

[0074] The display device 140 includes, for example, a fixed display device that is provided in a location that is easily visible to an operator seated in the cabin 10 and displays various information images to a user (operator) of the excavator SVL inside the cabin 10. The fixed display device is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0075] The display device 140 includes, for example, a portable display device that can be carried by a user, as described above. The portable display device includes, for example, a wearable display device that can display image information on a transparent surface of glasses or goggles that can be worn by a user, as described above. The wearable display device can display image information superimposed on the field of view of the user, on the premise that the user can view the surrounding situation from the transparent surface. Data regarding the position and attitude of the wearable display device is taken into the control device 110 through a short-distance communication line based on a wireless communication standard such as Bluetooth or WiFi. This allows the control device 110 to grasp the direction of the viewpoint and field of view of the user (operator) wearing the wearable display device, and display content that matches the field of view on the transparent surface. The wearable display device is, for example, smart glasses or AR (Augmented Reality) goggles. The portable display device includes, for example, a display device mounted on a mobile terminal carried by a user, as described above.

[0076] The input device 150 is provided within the cabin 10 in the vicinity of an operator seated therein, and receives various inputs from the operator. Signals corresponding to the received inputs are input to the control device 110.

[0077] For example, the input device 150 is an operation input device that accepts operation input. The operation input device includes a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), and the like.

[0078] Furthermore, for example, the input device 150 may be a voice input device that accepts voice input from an operator. The voice input device includes, for example, a microphone.

[0079] Furthermore, for example, the input device 150 may be a gesture input device that accepts a gesture input by an operator. The gesture input device includes, for example, an imaging device (an interior camera) installed in the cabin 10.

[0080] <<Communications>> As shown in FIGS. 2 and 3, the communication system of the shovel SVL according to this embodiment includes a communication device 115.

[0081] The communication device 115 is connected to the communication line NW and communicates with devices provided separately from the shovel SVL (for example, the position and orientation detection sensor 130 and a remote operation support device, etc.). The devices provided separately from the shovel SVL may include devices outside the shovel SVL as well as portable terminal devices brought into the cabin 10 by the user of the shovel SVL. The communication device 115 may be, for example, 4G (4 th Generation) and 5G (5 th The communication device 115 may include a mobile communication module conforming to standards such as the IEEE 802.11 Generation (IEEE 802.11b) standard. The communication device 115 may also include a satellite communication module, for example. The communication device 115 may also include a WiFi communication module, a Bluetooth communication module, or the like.

[0082] <<Control system>> As shown in FIGS. 2 and 3, the control system of the shovel SVL according to this embodiment includes a control device 110 and perimeter monitoring sensors 120A to 120G.

[0083] The control device 110 performs various controls related to the excavator SVL.

[0084] The control device 110 performs control related to the operation of a hydraulic actuator (driven element) of the excavator SVL, for example, with the above-mentioned operating hydraulic control valve as a control object.

[0085] For example, the control device 110 may perform control related to the operation of a hydraulic actuator (driven element) of the shovel SVL based on the operation of the operating device 26, with the operating hydraulic control valve being the control object.

[0086] Furthermore, for example, the control device 110 may perform control related to remote operation of a hydraulic actuator (driven element) of the excavator SVL, with an operation hydraulic control valve as a control target. Specifically, the control device 110 outputs a control signal corresponding to the content of remote operation specified by a remote operation signal received from an external device, to the operation hydraulic control valve, by the communication device 115 or the like. As a result, the control device 110 can supply a pilot pressure corresponding to the content of the remote operation from the operation hydraulic control valve to the control valve 17, thereby realizing the operation of the excavator SVL based on the remote operation by the operator.

[0087] Furthermore, for example, the control device 110 may perform control related to the automatic operation function of the excavator SVL by setting the operation hydraulic control valve as a control target. Specifically, the control device 110 outputs a control signal corresponding to an operation command related to the automatic operation function to the operation hydraulic control valve, regardless of whether the operation device 26 is operated or not. As a result, the control device 110 can supply pilot pressure corresponding to the operation command related to the automatic operation function from the operation hydraulic control valve to the control valve 17, thereby realizing the operation of the excavator SVL based on the automatic operation function.

[0088] The control device 110 also performs control related to the perimeter monitoring system 100, for example.

[0089] Note that some of the functions of the control device 110 may be realized by another controller (control device). That is, the functions of the control device 110 may be realized in a distributed manner by a plurality of controllers. For example, the control device 110 may be specialized in control related to the perimeter monitoring system 100, and control related to other excavator SVLs may be realized by another control device.

[0090] The periphery monitoring sensors 120A to 120C are imaging devices that capture images of the periphery of the excavator SVL and output image data relating to the situation in the periphery of the excavator SVL.

[0091] The perimeter monitoring sensor 120A, the perimeter monitoring sensor 120B, and the perimeter monitoring sensor 120C are attached to the upper rear end, the upper left end, and the upper right end of the upper rotating body 3, respectively, and capture images of the rear, left side, and right side of the upper rotating body 3. The perimeter monitoring sensor 120A captures an image range behind the upper rotating body 3, for example, an image range in the horizontal direction (i.e., the circumferential direction as seen from the shovel SVL) from the left rear to the right rear of the upper rotating body 3. The perimeter monitoring sensor 120B captures an image range on the left side of the upper rotating body 3, for example, an image range in the horizontal direction (the circumferential direction as seen from the shovel SVL) from the left front to the left rear of the upper rotating body 3. The perimeter monitoring sensor 120C captures an image range on the right side of the upper rotating body 3, for example, an image range in the horizontal direction (the circumferential direction as seen from the shovel SVL) from the right front to the right rear of the upper rotating body 3. Moreover, the perimeter monitoring sensors 120A to 120C may each be attached to the upper part of the upper revolving body 3 with an optical axis facing diagonally downward, and may capture an image in the vertical imaging range including the ground near the shovel SVL to the far side of the shovel SVL. Moreover, the perimeter monitoring sensors 120A to 120C may be configured so that their attitude state (vertical rotation state) is variable in response to an operation by a user (operator).

[0092] The perimeter monitoring sensors 120A to 120C output captured images (image data) at predetermined intervals (e.g., 1 / 30 seconds) from when the shovel SVL is started (i.e., key switch ON) to when it is stopped (i.e., key switch OFF). The captured images output from the perimeter monitoring sensors 120A to 120C are taken into the control device 110.

[0093] In addition, when the shovel SVL is remotely operated, in addition to the periphery monitoring sensors 120A to 120C, a periphery monitoring sensor 120X (for example, a front camera) may be provided that acquires an image for the remote-operating operator to check the situation in front of the shovel SVL.

[0094] The perimeter monitoring sensors 120D to 120G are distance sensors that output distance data, received light intensity data, etc., relating to objects in the perimeter of the excavator SVL. The following description will be centered on the case where the perimeter monitoring sensors 120D to 120G are LIDARs.

[0095] The perimeter monitoring sensors 120D, 120E, 120F, and 120G are respectively attached to the upper left rear end, the upper right rear end, the upper left end, and the upper right end of the upper rotating body 3. The perimeter monitoring sensors 120D, 120E, 120F, and 120G acquire information on the situations on the left rear, right rear, left side, and right side of the upper rotating body 3, respectively.

[0096] The perimeter monitoring sensors 120D to 120G may be, for example, a scanning LIDAR, a three-dimensional laser scanner capable of scanning the infrared laser irradiation direction in the vertical and horizontal directions. The perimeter monitoring sensors 120D to 120G may also be, for example, a so-called flash-type LIDAR that irradiates infrared rays from a light-emitting module over a wide three-dimensional range and captures the reflected light (infrared rays) with a three-dimensional distance image element.

[0097] The periphery monitoring sensor 120D is configured to be able to irradiate infrared rays to an irradiation range on the left rear side of the upper rotating body 3, for example, an irradiation range in the horizontal direction (i.e., the circumferential direction as seen from the shovel SVL) extending from the left rear to the rear of the upper rotating body 3. The periphery monitoring sensor 120E is configured to be able to irradiate infrared rays to an irradiation range on the right rear side of the upper rotating body 3, for example, an irradiation range in the horizontal direction (the circumferential direction as seen from the shovel SVL) extending from the right rear to the rear of the upper rotating body 3. The periphery monitoring sensor 120F is configured to be able to irradiate infrared rays to an irradiation range on the left side of the upper rotating body 3, for example, an irradiation range in the horizontal direction (the circumferential direction as seen from the shovel SVL) extending from the left front to the left rear of the upper rotating body 3. The periphery monitoring sensor 120G is configured to be able to irradiate infrared rays to an irradiation range on the right side of the upper rotating body 3, for example, an irradiation range in the horizontal direction (the circumferential direction as seen from the shovel) extending from the right front to the right rear of the upper rotating body 3. In addition, the perimeter monitoring sensors 120D to 120F are mounted on the upper part of the upper rotating body 3 so that their optical axes (i.e., the reference axis of the infrared radiation direction) face diagonally downward, and have an infrared radiation irradiation range in the vertical direction centered on the part of the ground relatively close to the excavator SVL.

[0098] The perimeter monitoring sensors 120D-120F each output light reception information at a predetermined cycle from start to stop of the shovel SVL. The data (distance data, light reception intensity data, etc.) output from the perimeter monitoring sensors 120D-120G is taken into the control device 110.

[0099] <Operations related to monitoring the situation around the excavator> 3, in this example, a position and orientation detection sensor 130 (in this example, an imaging device) is provided in the work site around the shovel SVL. The position and orientation detection sensor 130 may be one or more, as long as it is possible to acquire data related to the positions and orientations of all of the periphery monitoring sensors 120A to 120G of the shovel SVL.

[0100] The position and attitude detection sensor 130 transmits data relating to the position and attitude of each of the periphery monitoring sensors 120A to 120G to the excavator SVL via a predetermined communication line.

[0101] The control device 110 can receive and acquire output data from the position and orientation detection sensor 130 via the communication device 115 .

[0102] As described above, the control device 110 creates an integrated data group using the output data of the position and orientation detection sensor 130 based on the output data of the perimeter monitoring sensors 120A to 120G.

[0103] Furthermore, the control device 110 detects an object to be monitored (hereinafter, a "monitored object") within a predetermined range (hereinafter, a "monitored area") around the shovel SVL based on, for example, the integrated data group. The monitored area is, for example, a range within a predetermined distance (for example, 5 meters) from the shovel SVL. The monitored object includes, for example, a person such as a worker. The monitored object may also include, for example, a moving obstacle such as a work machine including another shovel or a work vehicle. The monitored object may also include, for example, a stationary obstacle such as a utility pole, a fence, or a hole.

[0104] In this case, the control device 110 may comprehensively detect monitored objects within the monitoring area based on the output data of the perimeter monitoring sensors 120A to 120C and the output data of the perimeter monitoring sensors 120D to 120G (or the converted data) contained in the integrated data group.

[0105] The control device 110 may use different data when detecting an object in the monitoring area and when determining the type of the object. For example, the control device 110 may detect an object in the monitoring area based on the output data (or the converted data) of the perimeter monitoring sensors 120D-120F included in the integrated data group. When an object is detected in the monitoring area, the control device 110 may recognize the type of the object in the image data based on the output data (image data) of the perimeter monitoring sensors 120A-120C included in the integrated data group. The control device 110 may then compare the position in the image data of the object whose type has been recognized with the position of the object detected in the monitoring area, and identify the type of the object detected in the monitoring area.

[0106] For example, when the control device 110 detects a monitored object in the monitoring area, the control device 110 outputs an alarm indicating the presence of the monitored object in the monitoring area to the operator, workers around the shovel SVL, etc., using a visual method or an audio method. Specifically, the control device 110 displays information indicating that the monitored object has been detected on the display device 140 in the cabin 10. The control device 110 may also output an alarm to the operator inside the cabin 10 or workers outside the cabin 10 using a speaker, a buzzer, or the like. This allows the operator inside the cabin 10 to recognize that a monitored object exists (has entered) in the monitoring area around the shovel SVL. In addition, the control device 110 can urge the workers around the shovel SVL to evacuate people as monitored objects from the periphery of the shovel SVL, or to evacuate obstacles other than people as monitored objects. In addition, when the shovel SVL is remotely operated, the control device 110 may transmit a signal corresponding to the alarm to the operator performing the remote operation through the communication device 115 to the remote operation support device. This allows the operator performing remote operation to recognize, via the remote operation display device of the remote operation support device, that a monitored object exists (has entered) in the monitoring area around the shovel SVL. Therefore, the perimeter monitoring system 100 can suppress a situation in which the shovel SVL comes into contact with the monitored object when the monitored object exists in the monitoring area around the shovel SVL. This can improve the safety of the shovel SVL.

[0107] Furthermore, the control device 110 may forcibly restrict the operation of the actuator (e.g., hydraulic actuator) of the shovel SVL when, for example, a monitored object is detected within the monitoring area. Specifically, the control device 110 may decelerate the operation of the actuator of the shovel SVL in response to the operation of the operator or an operation command corresponding to the automatic driving function, or may stop the operation of the actuator regardless of the presence or absence of the operation. More specifically, the control device 110 may change a control signal for an operating hydraulic control valve or invalidate the operation signal, for example, in response to an operation signal from the electric operating device 26 so that the operating speed of the actuator becomes relatively slow. In this way, the periphery monitoring system 100 can suppress a situation in which the shovel SVL comes into contact with the monitored object when the monitored object is present within the monitoring area around the shovel SVL. This can improve the safety of the shovel SVL.

[0108] Further, the control device 110 may display image information representing the situation around the shovel SVL on the display device 140 based on, for example, the integrated data group. Specifically, the control device 110 may display image information representing the situation around the shovel SVL on the display device 140 based on the integrated data group acquired in real time. This allows users such as operators and supervisors who monitor the work, etc., of the shovel SVL from outside the shovel SVL to monitor the situation around the shovel SVL in real time. Further, the control device 110 may display image information representing the situation around the shovel SVL on the display device 140 based on, for example, the integrated data group in time series stored in a predetermined storage device. This allows the user to check the situation around the shovel SVL after the fact and to verify the safety of the work site, etc.

[0109] For example, Fig. 4 is a diagram showing an example of the display contents of the display device 140. Specifically, Fig. 4 is a diagram showing a specific example of the display contents of the above-mentioned stationary display device.

[0110] In this example, the display contents of the stationary display device when a worker is present within the monitoring area behind the excavator SVL (upper rotating body 3) are shown.

[0111] As shown in FIG. 4, image information 400 showing the situation around the excavator SVL is displayed on the display device 140 (fixed display device).

[0112] The image information 400 displays a shovel image CG and a surrounding image EP showing the surroundings of the shovel SVL.

[0113] The shovel image CG shows a schematic representation of the shovel SVL as seen from directly above.

[0114] The surrounding image EP is a viewpoint conversion image generated based on the image data of each of the surrounding monitoring sensors 120A to 120C included in the integrated data group. Specifically, the control device 110 generates the surrounding image EP as a viewpoint conversion image by performing viewpoint conversion processing, synthesis processing, and the like on the image data of each of the surrounding monitoring sensors 120A to 120C included in the integrated data group. This allows the user (operator) to grasp the situation around the excavator SVL through the surrounding image EP. In addition, the surrounding image EP is arranged around the excavator image CG so as to match the positional relationship between the acquisition range (imaging range) of the image data of the surrounding monitoring sensors 120A to 120C and the excavator SVL (specifically, the upper rotating body 3). This allows the user to easily grasp the positional relationship between the object reflected in the surrounding image EP and the excavator SVL.

[0115] The surrounding image EP is composed of a combination of an overhead image BVP of the surrounding area adjacent to the shovel SVL viewed from directly above, and a horizontal image HVP of the surrounding area viewed horizontally from the shovel SVL, which is arranged around the overhead image BVP. This allows the user (operator) to check the surrounding situation close to the shovel SVL by viewing the shovel SVL and its adjacent surrounding area from directly above the shovel SVL through the surrounding image EP. The surrounding image EP is obtained by projecting the image data of each of the surrounding monitoring sensors 120A to 120C onto a space model, and then reprojecting the projected image projected onto the space model onto another two-dimensional plane. The space model is a projection target of the captured image in the virtual space, and is composed of one or more planes or curved surfaces including a plane or curved surface other than the plane on which the captured image is located.

[0116] It should be noted that a synthesized viewpoint conversion image may be generated and displayed on the display device 140 based on the output data of some of the perimeter monitoring sensors 120A to 120C (that is, two perimeter monitoring sensors 120X).

[0117] As described above, since the worker is present behind the shovel SVL, the worker 401 is shown in a position in the peripheral image EP that corresponds to the rear of the shovel image CG. This allows the operator to recognize that the worker is present in a relatively close position behind the shovel SVL by visually recognizing the worker 401 in the peripheral image EP.

[0118] Also, a frame-shaped marker 402 is displayed superimposed at a position in the peripheral image EP corresponding to the worker 401, specifically, at the foot position of the worker 401. For example, the control device 110 may recognize the position of an object in the periphery of the excavator SVL based on the output data (or the converted data) of the periphery monitoring sensors 120D-120G included in the integrated data. Then, the control device 110 may display the marker 402 at a position on the peripheral image corresponding to the position of an object recognized behind the excavator SVL (upper rotating body 3). In this way, the control device 110 can display on a stationary display device a peripheral image based on the output data of the periphery monitoring sensors 120A-120C and the marker 402 based on the output data of the periphery monitoring sensors 120D-120F based on the integrated data group.

[0119] The image information in FIG. 4 may be displayed on a portable display device instead of or in addition to the stationary display device. When the excavator SVL is remotely operated, the display device for remote operation of the remote operation support device displays the same display contents as those in FIG. 4. In this case, the image information 400 generated by the control device 110 may be transmitted to the remote operation support device through the communication device 115. A part or all of the functions of the control device 110 may be transferred to the remote operation support device. In this case, the output data of the periphery monitoring sensors 120A to 120G is transmitted from the communication device 115 to the remote operation support device through a predetermined communication line. The same applies below to the case where the display contents similar to those in FIG. 5 are displayed on a wearable display device worn by a remote operation operator.

[0120] Moreover, for example, Fig. 5 is a diagram showing another example of the display content of the display device 140. Specifically, Fig. 5 is a diagram showing a specific example of the display content of the transmissive surface of the above-mentioned wearable display device.

[0121] In this example, the display content of the wearable display device is displayed when an operator wearing the wearable display device enters the cabin 10 with a worker and a traffic cone present in the monitoring area to the right of the excavator SVL.

[0122] As shown in FIG. 5, the user can visually recognize the state inside the cabin 10 through the transparent surface of the wearable display device.

[0123] Specifically, an operating device 26 and a driver's seat 28 are disposed in the cabin 10. A front window 10WF, a rear window 10WB and a right side window 10WR through which the outside can be seen are provided on the front, rear and right side of the cabin 10, respectively. A left side window is also provided on the left side (door) of the cabin 10. Through these windows, the user can directly see the front, rear, left and right of the excavator SVL as seen from the cabin 10.

[0124] Here, as shown in Figs. 2 and 3, the cabin 10 is mounted on the left front part of the upper rotating body 3, and the boom 4 is attached to the right side thereof, i.e., to the center of the front part of the upper rotating body 3, in a manner that allows it to be raised and lowered. Also, as shown in Fig. 2, the height of the house part of the upper rotating body 3 reaches about half the height of the cabin 10. Therefore, as shown in Fig. 5, the operator's field of view as seen from the right side window 10WR of the cabin 10 is largely occupied by the base end part of the boom 4 and the house part of the upper rotating body 3. Therefore, the operator cannot directly see through the right side window the objects (worker or triangular cone) present at the back of the base end part of the boom 4 and the house part of the upper rotating body 3, i.e., on the right side of the (upper rotating body 3) to the excavator SVL.

[0125] In contrast, in this example, the control device 110 displays the object icons 501, 502 superimposed so that the boom 4 and the worker and triangular cone located at the back of the house part of the upper rotating body 3 can be seen through the transparent surface of the wearable display device by the operator.

[0126] Specifically, the control device 110 specifies (estimates) the field of view on the reference coordinates that the operator sees through the transparent surface based on data on the position and posture of the control device 110 captured from the wearable display device through a short-distance communication line or the like. Then, when the control device 110 recognizes a monitored object that exists behind an obstruction included in the field of view of the operator seen through the transparent surface, the control device 110 displays an object icon corresponding to the monitored object that exists behind the obstruction in a transparent manner. More specifically, the control device 110 displays the object icon so that the object icon is displayed in a direction and size that the monitored object would appear in if there were no obstruction. In this example, the obstruction corresponds to the boom 4, the upper rotating body 3, the cabin 10, and the like. Also, in this example, the monitored object corresponds to a person and a triangular cone. This allows the user to grasp the presence of an obstacle that cannot be seen in the direct field of view behind the obstruction in a manner that the direction and size of the object seen from the user's perspective are substantially the same as when seen in the actual direct field of view. Therefore, the user can easily grasp the presence of the obstacle when starting work with the excavator SVL.

[0127] Naturally, even when the user (operator) is seated in the cockpit 28 of the cabin 10 and faces to the right, an object icon corresponding to a monitored object recognized at the base end of the boom 4 or at the back of the house of the upper rotating body 3 is displayed on the transparent surface of the wearable display device. This allows the user to easily grasp the presence and position of an obstacle that cannot be seen directly while operating the excavator SVL.

[0128] In addition, instead of being displayed in the user's field of view as seen from the transparent surface of the wearable display device, the above-mentioned object icon may be displayed on a display device (e.g., a combiner) installed on a transparent surface such as the right side window 10WR of the cabin.

[0129] Furthermore, text information indicating the type of monitored object ("person" and "triangle cone") corresponding to each of the object icons 501 and 502 is displayed superimposed thereon. This allows the user to recognize that the obstacles corresponding to the object icons 501 and 502 are a person such as a worker and a triangular cone.

[0130] Furthermore, the shapes of the object icons 501 and 502 may be configured to typically represent a "person" or a "traffic cone."

[0131] When the shovel SVL is remotely operated, the wearable display device worn by an operator who performs remote operation using a remote operation support device may also display the same display content as in Fig. 5. In this case, image information showing the surrounding situation as seen from the shovel SVL (for example, an image captured by a front camera of the shovel SVL) may be displayed on the wearable display device of the operator who performs remote operation, and an object icon or the like may be superimposed on the image information.

[0132] Also, for example, under the control of the control device 110, the display device 140 may display image information showing the state of the periphery of the shovel SVL in three dimensions while freely changing the viewpoint in response to the user's operation through the input device 150. This allows the user to grasp the state of the periphery of the shovel SVL from any viewpoint. Therefore, the user can more easily monitor, for example, the presence of an object in a location corresponding to the blind spot of the operator of the cabin 10. Therefore, the convenience of the user can be improved, and the safety of the shovel SVL can be improved.

[0133] Specifically, the control device 110 may generate a viewpoint conversion image showing the surroundings of the shovel SVL as seen from an arbitrary viewpoint according to the user's operation, based on image data of the imaging device as the periphery monitoring sensor 120X included in the integrated data. The control device 110 may also generate a composite viewpoint conversion image showing the surroundings of the shovel SVL as seen from an arbitrary viewpoint according to the user's operation, based on image data of a plurality of imaging devices. Then, the control device 110 may cause the display device 140 to display three-dimensional image information showing the surroundings of the shovel SVL, generated by the viewpoint conversion process or the like.

[0134] Furthermore, the control device 110 may generate image information composed of a point cloud corresponding to objects in the periphery of the shovel SVL as viewed from an arbitrary viewpoint according to the user's operation, based on point cloud data of a distance sensor as the periphery monitoring sensor 120X included in the integrated data. Then, the control device 110 may cause the display device 140 to display three-dimensional image information composed of the generated point cloud. In this case, the control device 110 may simultaneously display three-dimensional data (e.g., polygon data) corresponding to the shovel SVL that is prepared in advance. In this way, the display device 140 can allow the user to appropriately recognize the sense of distance between the shovel SVL and the surrounding objects by using the point cloud corresponding to the objects in the periphery of the shovel SVL.

[0135] Further, the control device 110 may, for example, highlight a portion (for example, a point cloud portion) corresponding to the monitored object on the image information three-dimensionally showing the surroundings of the shovel SVL displayed on the display device 140. Specifically, the control device 110 may display a marker (for example, a frame surrounding the portion) at a location corresponding to the monitored object on the image information of the display device 140, change the color, or blink the color. This allows the user to properly grasp the presence and position of the monitored object on the image information three-dimensionally showing the surroundings of the shovel SVL displayed on the display device 140. In this case, the control device 110 may recognize (detect) the monitored object based on the output data of the captured image as the periphery monitoring sensor 120X. This is because the control device 110 can more properly recognize (detect) the monitored object through image recognition based on the output data of the captured image even in a situation where it is difficult to identify the type of object around the shovel SVL only from the output data of the distance sensor, for example.

[0136] [Example of application of perimeter monitoring system to continuous unloader] Next, with reference to Figs. 6 to 9, application of the perimeter monitoring system 100 to a continuous unloader ULD will be described. For example: We will explain about this.

[0137] <Configuration of continuous unloader> Fig. 6 to Fig. 9 are diagrams showing an example of an application example of the perimeter monitoring system 100 to a continuous unloader ULD. Specifically, Fig. 6 is an overall view showing an example of a continuous unloader ULD. Fig. 7 to Fig. 9 are diagrams showing a front view, a side view, and a cross-sectional view of a bucket elevator 59, respectively.

[0138] As shown in Fig. 6, the continuous unloader ULD (an example of a work machine) is a so-called bucket elevator type that is installed at a quay QY and continuously carries (unloads) onto land bulk cargo M in a hold HD of a ship SP berthed at the quay QY. The bulk cargo M is, for example, coal, coke, iron ore, etc.

[0139] The quay QY is constructed of, for example, reinforced concrete, and two rails 53 are installed on the quay QY in a direction parallel to the extension of the quay QY, i.e., the longitudinal direction of the ship SP to be docked. The continuous unloader ULD is configured to be movable on the two rails 53, and unloads cargo from the ship SP while stopped at a predetermined position.

[0140] The continuous unloader ULD includes a running section 52, a rotating frame 55, a boom 57, a bucket elevator 59, and a cab 66.

[0141] The running part 52 is configured to be movable on two rails 53 of the quay QY.

[0142] The rotating frame 55 is mounted on the running part 52 so as to be capable of rotating.

[0143] The boom 57 is provided to extend forward from the revolving frame 55 to the portion on the sea surface where the ship SP is to be docked from the quay QY, and is configured to be capable of being raised and lowered relative to the revolving frame 55. Specifically, the boom 57 can be raised and lowered in response to the extension and contraction of a cylinder 65 attached between the boom 57 and the revolving frame 55.

[0144] The bucket elevator 59 is provided at the tip of the boom 57 so as to extend downward, i.e., toward the ship SP (hold HD). As shown in Fig. 6 to Fig. 9, the bucket elevator 59 has a scraping part 61 at its tip, and bulk goods M scraped by the scraping part 61 are transported upward by the bucket 77 and landed on land.

[0145] 8, the scraping unit 61 can change its inclination angle in the front-rear direction with respect to the elevator body 64. For example, when scraping off loose cargo M directly below the opening OP of the hold HD, the scraping unit 61 is used in a state where it is not inclined in the front-rear direction with respect to the elevator body 64 (as shown by the solid line in the figure). On the other hand, when scraping off loose cargo M that is present on the inner wall side of the opening OP of the hold HD, the scraping unit 61 is used in a state where it is inclined forward with respect to the elevator body 64 (as shown by the dashed dotted line in the figure).

[0146] A parallel link 58 is provided between the revolving frame 55 and the bucket elevator 59, and the action of the parallel link 58 allows the bucket elevator 59 to maintain a vertical state regardless of the hoisting angle of the boom 57. The bucket elevator 59 can also move up and down in response to the up and down hoisting of the boom 57. A counterweight 63 is supported on the revolving frame 55 via a link extending rearward on the opposite side to the boom 57, and a balancing lever 62 is provided to connect the counterweight 63 and the boom 57. This allows the load to be balanced between the bucket elevator 59 and the counterweight 63.

[0147] The operator's cab 66 is provided at the front of the rotating frame 55 (i.e., at the position in the direction in which the boom 57 extends) and is used for an operator to sit in and operate the continuous unloader ULD.

[0148] In the cab 66, for example, a control device 110 is installed. In addition, in the cab 66, a display device 140 and an input device 150 are provided.

[0149] When the continuous unloader ULD is used to unload bulk cargo M, an operator is placed in the cab 66, and a worker is placed inside the hold HD to inform the operator of the status of the bulk cargo. The operator in the cab 66 and the worker in the hold HD work together to unload the bulk cargo M using the continuous unloader ULD.

[0150] The operator in the cab 66 can operate the continuous unloader ULD while checking the image information displayed on the display device 140, which shows the opening OP of the hold HD and the condition inside it, as well as the contents of communications from workers about the condition inside the hold HD.

[0151] As shown in Figures 7 and 8, the bucket elevator 59 includes an elevator body 64 that is provided to extend in the vertical direction, and a chain bucket 79 that performs a circulating motion between an upper portion 59a and a lower portion (scraping portion 61) of the bucket elevator 59.

[0152] The chain bucket 79 includes a pair of roller chains 75 and a plurality of buckets 77 supported so as to be suspended from the pair of roller chains 75 .

[0153] The roller chain 75 passes through the inside of the elevator body 64 and is endlessly connected in a circulating manner between the upper part 59a and the lower part (scraping part 61) of the bucket elevator 59.

[0154] Bucket elevator 59 also includes a drive roller 81 a around which roller chain 75 is stretched, driven rollers 81 b and 81 c that guide roller chain 75 , and a deflecting roller 83 .

[0155] The drive roller 81a is provided on an upper portion 59a of the bucket elevator 59, and the driven rollers 81b, 81c are provided in the scraping portion 61 at a predetermined interval in the front-rear direction.

[0156] The diverting roller 83 is disposed below the drive roller 81 a in the upper portion 59 a of the bucket elevator 59 , and is configured to be able to divert the traveling direction of the roller chain 75 .

[0157] A cylinder 85 is connected between the driven rollers 81b, 81c. The distance between the axes of the driven rollers 81b, 81c is varied in response to the extension and contraction of the cylinder 85, and as a result, the circumferential path of the movement of the chain bucket 79 is varied.

[0158] The roller chain 75 is driven by a driving roller 81a and moves in an orbital motion in the direction of an arrow W relative to the elevator body 64. The chain bucket 79 circulates between the upper portion 59a of the bucket elevator 59 and the scraping portion 61 while moving in an orbital motion.

[0159] When the roller chain 75 moves in a substantially horizontal direction from the driven roller 81b to 81c in the scraping section 61, the bucket 77 scrapes the bulk goods M from the opening of the bucket 77 into the inside thereof. The bucket 77 that has scraped and stored the bulk goods M rises with its opening facing upward as the roller chain 75 rises from the driven roller 81c to the driving roller 81a. When the bucket 77 arrives at the upper part 59a of the bucket elevator 59, the opening of the bucket 77 turns downward as the roller chain 75 changes direction from upward to downward as it passes the driving roller 81a. As a result, the bulk goods M in the bucket 77 are sent from the discharge chute to a rotary feeder 87 provided on the outer periphery of the bucket elevator 59.

[0160] The rotary feeder 87 transports the bulk material M sent from the bucket 77 through a discharge chute to a boom conveyor 89 provided on the boom 57.

[0161] The boom conveyor 89 is disposed inside the boom 57. Bulk goods M on the rotary feeder 87 of the bucket elevator 59 are transferred to the boom conveyor 89 and transported toward the revolving frame 55. On the revolving frame 55 side, a hopper is provided at the end of the boom conveyor 89, and the bulk goods M transported by the boom conveyor 89 are supplied to a belt conveyor 93 through the hopper.

[0162] The belt conveyor 93 is provided on the traveling section 52. The belt conveyor 93 transports the bulk goods M to the ground belt conveyor 95. As a result, the bulk goods M are conveyed to the ground equipment 99 via the ground belt conveyor 95.

[0163] As shown in FIGS. 6 to 9, the continuous unloader ULD includes periphery monitoring sensors 120A to 120F and a position and orientation detection sensor .

[0164] The perimeter monitoring sensors 120A to 120D are imaging devices. The perimeter monitoring sensors 120A to 120D are provided at equal intervals in the circumferential direction on the outer circumferential surface of the upper portion 59a of the bucket elevator 59. The perimeter monitoring sensors 120A to 120D are arranged with their optical axes facing downward so as to be able to capture an image of an opening OP of a hold HD of the ship SP from above.

[0165] The periphery monitoring sensors 120E, 120F are imaging devices. The periphery monitoring sensors 120E, 120F are provided on the left and right outer surfaces of the fixed part of the lower part (scraping part 61) of the bucket elevator 59. The periphery monitoring sensors 120E, 120F are arranged so that their optical axes are directed downward and can capture images of the inside of the hold HD (i.e., the top of the bulk cargo M) from above.

[0166] The periphery monitoring sensors 120A to 120F output captured images (image data) at predetermined intervals (e.g., 1 / 30 seconds) from the start to the stop of the continuous unloader ULD. The captured images output from the periphery monitoring sensors 120A to 120D are taken into the control device 110.

[0167] Further, the continuous unloader ULD (bucket elevator 59) may be provided with other types of perimeter monitoring sensors 120X (e.g., distance sensors) in addition to the perimeter monitoring sensors 120A-120F (imaging devices). For example, one or more distance sensors may be provided as the perimeter monitoring sensor 120X capable of acquiring point cloud data of a range including the imaging ranges of the perimeter monitoring sensors 120A-120D. Furthermore, one or more distance sensors may be provided as the perimeter monitoring sensor 120X capable of acquiring point cloud data of a range including the imaging ranges of the perimeter monitoring sensors 120E and 120F.

[0168] The position and orientation detection sensor 130 acquires data relating to the positions and orientations of the perimeter monitoring sensors 120A to 120F. The position and orientation detection sensor 130 is, for example, an imaging device.

[0169] The position and orientation detection sensor 130 includes position and orientation detection sensors 130A to 130D.

[0170] The position and orientation detection sensors 130A and 130B are provided on the outer peripheral surface of the lower part of the elevator main body 64, and acquire data on the positions and orientations of the periphery monitoring sensors 120A, 120B and the periphery monitoring sensors 120C, 120D, respectively. In addition, the position and orientation detection sensors 130A and 130B may acquire data on the positions and orientations of one or more distance sensors serving as the periphery monitoring sensor 120X capable of acquiring point cloud data of a range including the imaging ranges of the periphery monitoring sensors 120A to 120D, for example.

[0171] The position and orientation detection sensors 130C and 130D are provided at approximately the same positions as the position and orientation detection sensors 130A and 130B on the outer peripheral surface of the lower part of the elevator main body 64. The position and orientation detection sensors 130C and 130D acquire data on the positions and orientations of the periphery monitoring sensors 120E and 120F, respectively. The position and orientation detection sensors 130C and 130D may also acquire data on the positions and orientations of one or more distance sensors serving as the periphery monitoring sensor 120X capable of acquiring point cloud data of a range including the imaging ranges of the periphery monitoring sensors 120E and 120F, for example.

[0172] The output data of the position and orientation detection sensors 130A to 130D is input to the control device 110.

[0173] <Operation to monitor the conditions around the continuous unloader (bucket elevator)> As described above, the control device 110 creates an integrated data group based on the output data of the perimeter monitoring sensors 120A to 120F, using the output data of the position and orientation detection sensors 130A to 130D.

[0174] The control device 110 may also cause the display device 140 to display image information showing the situation around the bucket elevator 59, for example, based on the integrated data group. Specifically, the control device 110 may cause the display device 140 to display image information showing the situation around the bucket elevator 59, based on the integrated data group acquired in real time. This allows users such as operators and supervisors who monitor the work, etc., from outside the continuous unloader ULD to monitor the situation around the bucket elevator 59 in real time. The control device 110 may also cause the display device 140 to display image information showing the situation around the bucket elevator 59, for example, based on the integrated data group in time series stored in a predetermined storage device. This allows the user to check the situation around the bucket elevator 59 after the fact and verify the safety of the site, etc.

[0175] For example, the control device 110 displays image information (hereinafter, "hold opening image") showing the opening OP of the hold HD as viewed from the upper portion 59a of the bucket elevator 59, based on the output data of the perimeter monitoring sensors 120A to 120D included in the integrated data group.

[0176] In this case, the control device 110 may synthesize the output data (image data) of the periphery monitoring sensors 120A to 120D with the bucket elevator 59 as the reference (center), and display the synthesized image on the display device 140. This allows the operator to easily confirm the positional relationship between the bucket elevator 59 and the outer edge of the opening OP over the entire circumference through a single piece of image information (synthetic image).

[0177] Furthermore, the control device 110 may perform known viewpoint conversion processing and synthesis processing on the image data of the periphery monitoring sensors 120A-120D to cause the display device 140 to display the hold opening image as a viewpoint conversion image (bird's-eye view image) of the opening OP viewed from directly above. This allows the operator to more easily grasp the positional relationship between the bucket elevator 59 and the outer edge of the opening OP from the bird's-eye view image of the opening OP viewed from directly above.

[0178] Also, for example, the control device 110 causes the display device 140 to display image information (hereinafter, "hold interior image") showing the situation around the scraping section 61 inside the hold HD based on the output data of the surrounding monitoring sensors 120E, 120F included in the integrated data group.

[0179] In this case, the control device 110 may synthesize the output data (image data) of the perimeter monitoring sensors 120E, 120F with the scraping unit 61 as the reference (center) and display the synthesized image on the display device 140. This allows the operator to easily check the condition of the bulk cargo M around the scraping unit 61 and the positional relationship between the scraping unit 61 and the inner wall of the hold HD over the entire circumference through a single image information (synthetic image).

[0180] Furthermore, the control device 110 may perform known viewpoint conversion processing and synthesis processing on the image data of the periphery monitoring sensors 120E, 120F to cause the display device 140 to display an image of the inside of the hold HD as a viewpoint conversion image (bird's-eye image) of the inside of the hold HD seen from directly above. This allows the operator to easily grasp the positional relationship between the tip of the bucket elevator 59 (scraping unit 61) and the surrounding bulk goods M, the inner wall of the hold HD, the workers inside the hold HD, and the like, from the bird's-eye image of the inside of the hold HD seen from directly above.

[0181] Also, for example, the control device 110 may cause the display device 140 to display both the opening image information and the hold interior image information based on the integrated data group.

[0182] In this case, the control device 110 may cause the hold opening image information and the hold interior image information to be displayed separately in different display areas (displays) of the display device 140.

[0183] Furthermore, the control device 110 may display the hold opening image and the hold interior image in the same display area of ​​the display device 140. For example, the control device 110 may display the hold opening image as an overhead image on the display device 140, and may superimpose and display the hold interior image as an overhead image so as to make the back of the opening OP of the hold opening image and the hull part of the ship SP outside the outer edge of the opening OP transparent. This allows the user (operator) of the cab 66 to check both the positional relationship between the opening of the hold HD and the bucket elevator 59 (elevator main body 64) and the situation around the bucket elevator 59 inside the hold HD in one display area of ​​the display device 140. Therefore, the control device 110 can improve the convenience for the user.

[0184] Also, for example, under the control of the control device 110, the display device 140 may display image information showing the situation around the bucket elevator 59 in three dimensions while freely changing the viewpoint in response to a user's operation through the input device 150. This allows the user to grasp the situation around the bucket elevator 59 from any viewpoint. Therefore, for example, by changing the viewpoint of one piece of image information, the user can easily grasp the situation around the bucket elevator 59 near the opening OP of the hold HD and the situation around the bucket elevator 59 inside the hold HD. This can improve the convenience for the user and the safety of the continuous unloader ULD.

[0185] Specifically, the control device 110 may generate a viewpoint conversion image showing the surroundings of the bucket elevator 59 as seen from an arbitrary viewpoint according to the user's operation, based on image data of the periphery monitoring sensors 120A-120F (imaging devices) included in the integrated data. The control device 110 may also generate a composite viewpoint conversion image showing the surroundings of the bucket elevator 59 as seen from an arbitrary viewpoint according to the user's operation, based on image data of the multiple imaging devices. Then, the control device 110 may cause the display device 140 to display three-dimensional image information showing the surroundings of the bucket elevator 59, generated by viewpoint conversion processing or the like.

[0186] Furthermore, the control device 110 may generate image information composed of a point cloud corresponding to objects in the vicinity of the bucket elevator 59 as viewed from an arbitrary viewpoint according to a user's operation, based on point cloud data of a distance sensor serving as the periphery monitoring sensor 120X included in the integrated data. Then, the control device 110 may cause the display device 140 to display three-dimensional image information composed of the generated point cloud. In this case, the control device 110 may simultaneously display three-dimensional data (e.g., polygon data) corresponding to the bucket elevator 59 that is prepared in advance. In this way, the display device 140 can allow the user to appropriately recognize the sense of distance to the objects in the vicinity of the bucket elevator 59 by using the point cloud corresponding to the objects in the vicinity of the bucket elevator 59.

[0187] Further, the control device 110 may, for example, highlight a portion (for example, a point cloud portion) corresponding to the monitored object on the image information three-dimensionally showing the state of the periphery of the bucket elevator 59 displayed on the display device 140. Specifically, the control device 110 may display a marker (for example, a frame surrounding the portion) at a location corresponding to the monitored object on the image information of the display device 140, change the color, or blink the color. The monitored object is, for example, a person corresponding to a worker inside the hold HD, an opening OP of the hold HD, an inner wall of the hold HD, etc. This allows the user to properly grasp the presence and position of the monitored object on the image information three-dimensionally showing the state of the periphery of the bucket elevator 59 displayed on the display device 140. In this case, the control device 110 may recognize (detect) the monitored object based on the output data of the periphery monitoring sensors 120A to 120F (captured images). This is because, for example, even in a situation where it is difficult to identify the type of object around the bucket elevator 59 from the output data of the distance sensor alone, the control device 110 can more appropriately recognize (detect) the monitored object through image recognition based on the output data of the captured image.

[0188] In addition, when the continuous unloader ULD is remotely operated, the same image information as that of the display device 140 is also displayed on the remote operation display device of the remote operation support device. In this case, the image information generated by the control device 110 may be transmitted to the remote operation support device through a communication device. Also, a part or all of the functions of the control device 110 may be transferred to the remote operation support device. In this case, the output data of the periphery monitoring sensor 120X including the periphery monitoring sensors 120A to 120F is transmitted to the remote operation support device from the communication device of the continuous unloader ULD through a predetermined communication line.

[0189] Furthermore, the control device 110 may perform known viewpoint conversion processing and synthesis processing on the output data of the periphery monitoring sensors 120E, 120F to display a viewpoint conversion image (bird's-eye view image) of the interior of the hold HD around the scraping unit 61 viewed from directly above on the display device 140. This allows the operator to more easily grasp the state of the bulk cargo M around the scraping unit 61 and the positional relationship between the scraping unit 61 and the inner wall of the hold HD through the bird's-eye view image of the interior of the hold HD viewed from directly above (the top of the scraping unit 61).

[0190] Furthermore, the control device 110 detects an obstacle within a predetermined range (monitoring area) around the bucket elevator 59, for example, based on the integrated data group. The obstacle is, for example, the outer edge of the opening OP of the hold HD or the inner wall of the inside of the hold HD. The monitoring area is, for example, a range within a predetermined distance (for example, several meters) from the bucket elevator 59 (the elevator body 64 and the scraping part 61).

[0191] In this case, the control device 110 may detect an obstacle in a monitoring area around the elevator main body 64 based on the output data (image data) of the perimeter monitoring sensors 120A-120D included in the integrated data group. The control device 110 may also detect an obstacle in a monitoring area around the scraping unit 61 based on the output data (image data) of the perimeter monitoring sensors 120E, 120F included in the integrated data group.

[0192] For example, when the control device 110 detects an obstacle in the monitoring area, it outputs an alarm to the operator or the like by using a visual method or an audio method, indicating that the bucket elevator 59 is too close to the outer edge of the opening OP of the hold HD or the inner wall of the hold HD. Specifically, the control device 110 causes the display device 140 in the driver's cab 66 to display information indicating the output of the alarm. The control device 110 may also output an alarm to the operator inside the driver's cab 66 using a speaker, a buzzer, or the like. This allows the operator inside the driver's cab 66 to. In addition, when the continuous unloader is remotely operated, the control device 110 may transmit a signal corresponding to the alarm to the remote operation support device through the communication device. This allows the remote operation support device to make the remote operation operator aware of the situation in which the bucket elevator 59 is too close to the outer edge of the opening OP of the hold HD or the inner wall of the hold HD through the remote operation display device or the like of the remote operation support device. Therefore, the perimeter monitoring system 100 can suppress the occurrence of a situation in which the bucket elevator 59 comes into contact with the ship SP. Therefore, the safety of the continuous unloader ULD can be improved.

[0193] Furthermore, the control device 110 may forcibly restrict the operation of the actuator of the continuous unloader when it detects, for example, an obstacle (such as the outer edge of the opening OP of the hold HD or the inner wall of the hold HD) within the monitoring area. Specifically, the control device 110 may slow down the operation of the actuator of the shovel SVL in response to the operation of the operator, or may stop the operation of the actuator regardless of the presence or absence of the operation. This allows the periphery monitoring system 100 to suppress an incident in which the bucket elevator 59 comes into contact with the ship SP. This allows the safety of the shovel SVL to be improved.

[0194] [Effect] Next, the operation of the perimeter monitoring system 100 according to this embodiment will be described.

[0195] In this embodiment, the perimeter monitoring system 100 includes a plurality of perimeter monitoring sensors 120X and a position and orientation detection sensor 130. Specifically, each of the plurality of perimeter monitoring sensors 120X is mounted on a work machine and acquires data related to the situation around the work machine. The position and orientation detection sensor 130 acquires data related to the position and orientation of each of the plurality of perimeter monitoring sensors 120X.

[0196] This allows the perimeter monitoring system 100 to accurately grasp the positions and orientations of the multiple perimeter monitoring sensors 120X using the position and orientation detection sensor 130. Therefore, the perimeter monitoring system 100 can more easily monitor the situation around the work machine, as described above. Also, the perimeter monitoring system 100 can more appropriately recognize the situation around the work machine, as described above.

[0197] In addition, in this embodiment, the perimeter monitoring system 100 may be equipped with a control device 110 that integrates data regarding the situation around the work machine acquired by each of the multiple perimeter monitoring sensors 120X based on data acquired by the position and orientation detection sensor 130.

[0198] This allows the periphery monitoring system 100 to integrate output data from the multiple periphery monitoring sensors 120X according to a unified standard (reference coordinate system), thereby allowing the periphery monitoring system 100 to monitor the situation around the work machine more easily and with greater accuracy.

[0199] Furthermore, in this embodiment, the periphery monitoring system 100 may include a display device 140 that displays image information showing the situation around the work machine based on data integrated by the control device 110 (integrated data group).

[0200] This enables the periphery monitoring system 100 to use the integrated data group to more easily and more accurately provide the user with image information that represents the situation around the work machine.

[0201] Furthermore, in this embodiment, the display device 140 (wearable display device) may display an image of the surroundings as seen from the work machine, or image information showing obstacles around the work machine within the user's field of vision as seen from the work machine.

[0202] As a result, the perimeter monitoring system 100 can provide image information showing obstacles around the work machine by superimposing it on an image of the periphery of the work machine and on the user's field of vision. This allows the user to check the situation around the work machine and additional image information in the same field of vision. Therefore, the perimeter monitoring system 100 can improve user convenience and also improve the safety of the work machine.

[0203] Furthermore, in this embodiment, the display device 140 (wearable display device) may display, in a transparent manner, an image of the surroundings as viewed from the work machine, or image information showing an obstacle that is present behind an obstruction in the user's field of vision as viewed from the work machine.

[0204] This allows the perimeter monitoring system 100 to transparently display obstacles behind obstructions around the work machine that are visible to the user, allowing the user to intuitively grasp the presence of an obstacle behind the obstruction.

[0205] Furthermore, in this embodiment, the display device 140 may display image information that represents the situation around the work machine over the entire acquisition range of data from two or more of the multiple perimeter monitoring sensors 120X.

[0206] As a result, the perimeter monitoring system 100 can provide the user with information on the situation around the work machine across the data acquisition ranges (imaging ranges) of two or more perimeter monitoring sensors (imaging devices) in the form of a single piece of image information. Therefore, the perimeter monitoring system 100 can improve user convenience.

[0207] Furthermore, in this embodiment, the multiple perimeter monitoring sensors 120X may include a combination of perimeter monitoring sensors 120X that are configured such that at least one of the relative position and attitude changes with the operation of the work machine.

[0208] In this case, as described above, there is a possibility that data of overlapping ranges around the work machine may be acquired by two or more perimeter monitoring sensors 120X, or that the overlapping ranges may change, resulting in an erroneous recognition of the situation around the work machine.

[0209] In contrast, in this embodiment, the perimeter monitoring system 100 can use output data from the position and orientation detection sensor 130. Therefore, even if a state occurs in which two or more perimeter monitoring sensors 120X acquire data of overlapping ranges around the work machine, or the overlapping range of data changes, the perimeter monitoring system 100 can appropriately recognize the data of the overlapping range as data of the same range. Therefore, even in such a situation, the perimeter monitoring system 100 can more appropriately recognize the situation around the work machine.

[0210] In this embodiment, the plurality of perimeter monitoring sensors 120X may include two or more types of sensors.

[0211] This allows the periphery monitoring system 100 to comprehensively use output data from two or more types of sensors to more appropriately monitor the conditions around the work machine.

[0212] Furthermore, in this embodiment, the display device 140 may display image information that three-dimensionally depicts the surroundings of the work machine while changing the viewpoint in response to a user operation.

[0213] In this way, the display device 140 allows the user to monitor the surroundings of the work machine while freely changing the viewpoint. Therefore, the user can more easily monitor the surroundings of the work machine using multiple surrounding monitoring sensors.

[0214] Furthermore, in this embodiment, the display device 140 may display image information that shows the surroundings of the work machine in three dimensions and in a manner that allows the user to recognize the sense of distance from the work machine, while changing the viewpoint in response to user operation.

[0215] This allows the display device 140 to use three-dimensional image information to allow the user to recognize the sense of distance between the work machine and surrounding objects, making it easier for the user to monitor the situation around the work machine.

[0216] Furthermore, in this embodiment, the display device 140 may display the above image information in such a way that the user can identify a predetermined object on the image information.

[0217] This allows the display device 140 to recognize the presence of a specific object (for example, a person such as a worker) that requires attention around the work machine on three-dimensional image information, making it easier for the user to monitor the situation around the work machine.

[0218] Furthermore, in this embodiment, the image information displayed on the display device 140 may include a point cloud representing objects in the vicinity of the work machine based on output data from a distance sensor serving as the periphery monitoring sensor 120X mounted on the work machine.

[0219] This enables the display device 140 to allow the user to specifically recognize the sense of distance between the work machine and surrounding objects, using a point cloud based on the output data of the distance sensor.

[0220] Furthermore, in this embodiment, the display device 140 may highlight and display a point cloud portion that corresponds to a predetermined object on the above image information, based on output data from an imaging device serving as the periphery monitoring sensor 120X mounted on the work machine.

[0221] This allows the display device 140 to recognize a specific object around the work machine by using, for example, output data from the imaging device, and to emphasize a point cloud portion on the image information that corresponds to the position of the recognized specific object. Therefore, the display device 140 allows the user to specifically identify the specific object on the image information.

[0222] [Transformation / Change] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist described in the claims.

[0223] For example, in the above-described embodiment, the excavator SVL is configured such that all of the driven elements, such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6, are hydraulically driven, but some or all of them may be electrically driven by electric actuators. In other words, the perimeter monitoring system 100 may be applied to hybrid excavators, electric excavators, etc.

[0224] In the above-described embodiment, an application example of the perimeter monitoring system 100 to an excavator SVL or a continuous unloader ULD is disclosed, but the perimeter monitoring system 100 may be applied to other work machines. The other work machines may include, for example, a mobile crane, an automated guided vehicle (AGV), a gantry crane, etc. [Explanation of symbols]

[0225] 100 Perimeter Monitoring System 110 Control device 120, 120A~120G, 120X Periphery monitoring sensor 130, 130A, 130B Position and orientation detection sensor 140 Display device 150 Input Device SVL Excavator (Work Machine) ULD Continuous Unloader (Work Machine)

Claims

1. A plurality of surrounding monitoring sensors mounted on the work machine for acquiring data relating to a surrounding situation of the work machine; a position and orientation detection sensor for acquiring data relating to the position and orientation of each of the plurality of perimeter monitoring sensors; a control device that integrates data relating to a situation around the work machine acquired by each of the plurality of periphery monitoring sensors based on data acquired by the position and attitude detection sensor, the plurality of perimeter monitoring sensors includes a combination of two or more perimeter monitoring sensors configured such that at least one of a relative position and an attitude changes with an operation of the work machine; the control device integrates data relating to a situation around the work machine, which is acquired by each of the two or more perimeter monitoring sensors, based on data relating to the position and orientation of each of the two or more perimeter monitoring sensors acquired by the position and orientation detection sensor. Perimeter surveillance system.

2. a display device that displays image information representing a situation around the work machine based on the data integrated by the control device, The perimeter monitoring system according to claim 1 .

3. The display device displays an image of the surroundings viewed from the work machine, or the image information showing obstacles in the vicinity of the work machine within a field of view of a user viewing the surroundings from the work machine. The perimeter monitoring system according to claim 2 .

4. the display device displays, in a transparent manner, an image of the surroundings viewed from the work machine, or the image information showing the obstacle present behind an obstruction in the user's field of vision when the user views the surroundings from the work machine. The perimeter monitoring system according to claim 3 .

5. the display device displays the image information representing the situation around the work machine over the entire range of acquisition of data from some or all of the plurality of perimeter monitoring sensors. The perimeter monitoring system according to any one of claims 2 to 4.

6. the display device displays the image information, which shows the surroundings of the work machine in three dimensions and in a manner that allows the user to recognize the sense of distance from the work machine, while changing a viewpoint in response to a user's operation, and displays a predetermined object on the image information so that the user can identify it. The perimeter monitoring system according to any one of claims 2 to 5.

7. The plurality of perimeter monitoring sensors include a distance sensor and an imaging device, the image information includes a point cloud representing objects around the work machine based on output data of the distance sensor; the display device displays, based on the output data of the imaging device, a point cloud portion corresponding to the predetermined object in the image information in an emphasized manner. The perimeter monitoring system according to claim 6.

8. The plurality of perimeter monitoring sensors include two or more types of sensors. A perimeter monitoring system according to any one of claims 1 to 7.

Images