Work machine, information processing device, and program

Abstract

Provided is a technology that makes it possible to improve safety in the periphery of a work machine. A work machine 100 according to one embodiment of the present disclosure is provided with a control device 110 that: acquires information indicating the shape of a work target or information indicating the shapes of the work target and an object in the periphery of the work target; predicts, on the basis of said information, a change in the shape of the work target according to the motion of the work machine 100 or the shapes of the work target and the object in the periphery of the work target; and, on the basis of the prediction result of the change, gives notification to an operator or to an external entity, or restricts the motion of the work machine 100.

Description

Work machine, information processing device, and program

[0001] The present disclosure relates to work machines and the like.

[0002] Conventionally, there is known a technology that ensures safety around a work machine by alerting the operator or the outside or restricting the operation of the work machine when a monitored object (e.g., a person) is detected in a monitoring area around the work machine (see Patent Document 1).

[0003] JP 2017-101419 A

[0004] However, it is desirable to further improve safety around the work machine.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a technique that can improve safety around a work machine.

[0006] In order to achieve the above object, one embodiment of the present disclosure provides a work machine that is equipped with a control device that acquires information representing the shape of a work object, or information representing the shapes of the work object and objects surrounding the work object, predicts changes in the shape of the work object or the work object and objects surrounding the work object in response to the operation of the work machine based on that information, and issues a warning to an operator or to the outside, or restricts the operation of the work machine, based on the predicted changes.

[0007] Furthermore, in another embodiment of the present disclosure, there is provided an information processing device that acquires information representing the shape of a work target of a work machine, or information representing the shape of the work target and objects surrounding the work target, predicts a change in the shape of the work target or the work target and objects surrounding the work target in response to the operation of the work machine based on the information, and issues a warning to an operator or outside the work machine based on the predicted change, or restricts the operation of the work machine.

[0008] Furthermore, in yet another embodiment of the present disclosure, a program is provided that causes an information processing device to acquire information representing the shape of a work target of a work machine, or information representing the shapes of the work target and objects surrounding the work target, and, based on that information, predict a change in the shape of the work target, or the work target and objects surrounding the work target, in response to the operation of the work machine, and, based on the predicted change, issue an alert to an operator or outside the work machine, or restrict the operation of the work machine.

[0009] According to the above-described embodiment, safety around the work machine can be improved.

[0010] 1 is a diagram illustrating an example of an operation support system. FIG. 1 is a diagram illustrating an example of a configuration related to remote operation of a work machine. FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. FIG. 3 is a functional block diagram illustrating a first example of a functional configuration of the operation support system. FIG. 4 is a diagram explaining an example of a method for calculating a safety level. FIG. 5 is a flowchart schematically illustrating a first example of processing of the operation support system. FIG. 6 is a diagram illustrating an example of an observation target area. FIG. 7 is a functional block diagram illustrating a second example of a functional configuration of the operation support system. FIG. 8 is a flowchart schematically illustrating a second example of processing of the operation support system. FIG. 9 is a side view illustrating an example of a shovel. FIG. 10 is a top view illustrating an example of a shovel. FIG. 11 is a diagram illustrating a first example of a change in the shape of the soil and sand around the shovel due to an excavation operation of the shovel. FIG. 12 is a diagram illustrating a second example of a change in the shape of the soil and sand around the shovel due to an excavation operation of the shovel. FIG. 13 is a diagram illustrating a third example of a change in the shape of the soil and sand around the shovel due to an excavation operation of the shovel. FIG. 14 is a diagram illustrating a fourth example of a change in the shape of the soil and sand around the shovel due to an excavation operation of the shovel. FIG. 15 is a diagram illustrating an example of a continuous unloader. FIG. 16 is a diagram illustrating an example of a continuous unloader. 10A and 10B are diagrams showing an example of changes in the shape of bulk goods around the scraping section due to the scraping operation of the continuous unloader.

[0011] Hereinafter, an embodiment will be described with reference to the drawings.

[0012] [Outline of Operation Support System] An outline of the operation support system SYS according to this embodiment will be described with reference to FIGS. 1 and 2. FIG.

[0013] Fig. 1 is a diagram showing an example of an operation support system SYS. Fig. 2 is a diagram showing an example of a configuration related to remote operation of a work machine 100.

[0014] As shown in FIG. 1 , the operation support system SYS includes a work machine 100 , an information processing device 200 , and a sensor group 300 .

[0015] The operation support system SYS uses an information processing device 200 to cooperate with the work machine 100 and provide support related to the operation of the work machine 100.

[0016] The operation support system SYS may include one or more work machines 100 .

[0017] The work machine 100 drives the work device 125 by the actuator 120 and performs a predetermined operation using the work device 125, thereby performing a predetermined task that changes the shape of a work target within a work site.

[0018] The work machine 100 is, for example, a shovel SVL, which will be described later (see FIGS. 10 and 11). In this case, the work target of the work machine 100 is, for example, earth and sand in a predetermined area within the work site where the shovel SVL is located. In this case, the predetermined operation of the work machine 100 is, for example, the digging operation or earth removal operation of the shovel SVL, and the predetermined work of the work machine 100 is, for example, the digging operation of the shovel SVL or the loading of earth and sand onto a truck. The work machine 100 may also be a continuous unloader ULD, which will be described later (see FIGS. 16 to 19). In this case, the work target of the work machine 100 is, for example, bulk cargo M loaded inside the hold HD of a ship SP moored at the work site (port area). In this case, the predetermined operation of the work machine 100 is the scraping operation of the continuous unloader ULD, and the predetermined work of the work machine 100 is the unloading operation of the bulk cargo M onto land.

[0019] The work machine 100 is equipped with a communication device 180 and is capable of communicating with an information processing device 200 via a predetermined communication line NW.

[0020] The communication line NW may include, for example, a local area network (LAN) within a predetermined range including a work site. The communication line NW may also include a wide area network (WAN). Examples of wide area networks include mobile communication networks that terminate at base stations, satellite communication networks that use communication satellites, and the Internet. The communication line NW may also include, for example, short-range communication lines based on wireless communication standards such as Wi-Fi and Bluetooth (registered trademark).

[0021] For example, the work machine 100 performs a predetermined operation by operating a driven element in response to the operation of an operator on board the work machine 100. The driven element of the work machine 100 is driven by an actuator 120 and is a movable part of the work machine 100. The driven element of the work machine 100 is, for example, the lower traveling body 1 (crawlers 1CL, 1CR), upper rotating body 3, boom 4, arm 5, bucket 6, etc. of the excavator SVL described below. Furthermore, the driven element of the work machine 100 is, for example, the traveling part 52, rotating body 55, bucket elevator 59 (scraping part 61), etc. of the continuous unloader ULD described below.

[0022] Furthermore, instead of or in addition to being configured to be operable by an operator on board the work machine 100, the work machine 100 may be configured to be capable of being remotely operated (also referred to as "remote operation") from outside the work machine 100. When the work machine 100 is remotely operated, the operator's cab of the work machine 100 (for example, the cabin 10 of an excavator SVL or the operator's cab 66 of a continuous unloader ULD) may be unmanned. Furthermore, when the work machine 100 is exclusively for remote operation, the operator's cab may be omitted. The following description will be given on the assumption that operator operation includes at least one of operation of the operating device 130 by an operator in the cab and remote operation by an external operator.

[0023] 2 , remote operation includes a mode in which the work machine 100 is operated by operation input to an actuator 120 of the work machine 100 performed by a remote operation support device 400 that can communicate with the work machine 100 via a communication line NW. The remote operation support device 400 may be provided separately from the information processing device 200, or may be the information processing device 200.

[0024] The remote operation support device 400 is provided, for example, in a management center that externally manages the work of the work machine 100. The remote operation support device 400 may also be a portable operation terminal, in which case the operator can remotely operate the work machine 100 while directly checking the work status of the work machine 100 from the vicinity of the work machine 100.

[0025] For example, the work machine 100 transmits to the remote operation support device 400, via the communication device 180, an image (hereinafter, "remote operation image") that allows the remote operation support device 400 to check the work status of the work machine 100, including the periphery of the work device 125. The remote operation image may be an image captured by an imaging device included in the periphery monitoring sensor 140, which will be described later, or a processed image that is generated by processing the captured image. Furthermore, if the remote operation image is a processed image, the work machine 100 may transmit, via the communication device 180, an image output by the imaging device to the remote operation support device 400, and the remote operation support device 400 may generate the remote operation image by processing the captured image received from the work machine 100. This allows the remote operation support device 400 to display the remote operation image on its own display device (hereinafter, "remote operation display device"). Therefore, an operator using the remote operation support device 400 can remotely operate the work machine 100 while checking the remote operation image displayed on the remote operation display device. Furthermore, information images (hereinafter referred to as "remote operation information images") similar to the various information images displayed on the output device 160 (display device 162) in the driver's seat of the work machine 100 may be displayed on the remote operation display device. This allows the operator using the remote operation support device 400 to remotely operate the work machine 100 while understanding various conditions of the work machine 100, for example, by checking the content of the remote operation information image displayed on the remote operation display device. The work machine 100 may then operate the actuator 120 and drive the driven element in accordance with a remote operation signal that indicates the content of the remote operation, and that is received by the communication device 180 from the remote operation support device 400.

[0026] Remote operation may also include a mode in which the work machine 100 is operated by external voice input, gesture input, or the like to the work machine 100 by a person (e.g., a worker) in the vicinity of the work machine 100. Specifically, the work machine 100 recognizes voices uttered by nearby workers, gestures made by workers, or the like, through a voice input device (e.g., a microphone) or gesture input device (e.g., an imaging device) or the like mounted on the work machine 100. The work machine 100 may then operate the actuator 120 and drive the driven element in accordance with the content of the recognized voice, gesture, or the like.

[0027] Furthermore, the work machine 100 may automatically operate the actuators regardless of the operation by the operator. This allows the work machine 100 to realize a function for automatically operating at least a portion of the driven elements including the work implement 125, that is, a so-called "automatic driving function" or "machine control (MC) function."

[0028] The automatic driving function includes, for example, a semi-automatic driving function (an operation-assisted MC function). The semi-automatic driving function is a function that automatically operates driven elements other than the driven element that is the target of operation in accordance with the operation of the operator. In other words, the semi-automatic driving function is a function that automatically operates actuators 120 other than the actuator 120 that is the target of operation in accordance with the operation of the operator. Hereinafter, the operation of a driven element of the work machine 100 and the operation of the actuator 120 that drives the driven element have the same meaning. The automatic driving function may also include a fully automatic driving function (a fully automatic MC function). The fully automatic driving function is a function that automatically operates at least a portion of a plurality of driven elements, assuming there is no operation by the operator. When the fully automatic driving function is enabled in the work machine 100, the driver's seat of the work machine 100 may be unmanned. Furthermore, if the work machine 100 is dedicated to fully automatic driving, the driver's seat may be omitted. Furthermore, the semi-automatic driving function, fully automatic driving function, etc. include, for example, a rule-based automatic driving function. A rule-based automatic driving function is an automatic driving function in which the operation details of driven elements that are the subject of automatic driving are automatically determined in accordance with predefined rules. Furthermore, semi-automatic driving functions, fully automatic driving functions, etc. may also include autonomous driving functions. The autonomous driving function is an automatic driving function in which the work machine 100 autonomously makes various decisions, and the operation details of driven elements that are the subject of automatic driving are determined in accordance with the results of those decisions.

[0029] Furthermore, the work of the work machine 100 may be remotely monitored. In this case, a remote monitoring support device having the same functions as the remote operation support device 400 may be provided. The remote monitoring support device is, for example, the information processing device 200. This allows a monitor, who is a user of the remote monitoring support device, to monitor the status of the work of the work machine 100 while checking the surrounding image displayed on the display device of the remote monitoring support device. Furthermore, for example, if the monitor determines it is necessary from a safety standpoint, the monitor can intervene in the operation by the operator or automatic operation of the work machine 100 and bring the work machine 100 to an emergency stop by making a specified input using an input device of the remote monitoring support device.

[0030] The information processing device 200 communicates with the work machine 100 to cooperate with it and provide support for the operation of the work machine 100 .

[0031] The information processing device 200 is, for example, a server device or a management terminal device that is installed in a management office where the work machine 100 is installed at a work site, or in a management center that manages the operating status of the work machine 100 and is located in a location different from the work site of the work machine 100. The server device may be an on-premises server, a cloud server, or an edge server. The management terminal device may be, for example, a fixed terminal device such as a desktop personal computer (PC), or a portable terminal device (i.e., a mobile terminal) such as a tablet terminal, smartphone, or laptop PC. In the latter case, workers at the work site, supervisors who supervise the work, managers who manage the work site, etc. can carry the portable information processing device 200 and move around the work site. In the latter case, the operator can, for example, bring the portable information processing device 200 into the driver's seat of the work machine 100.

[0032] The information processing device 200, for example, acquires data representing the operating status from the work machine 100. This enables the information processing device 200 to grasp the operating status of the work machine 100 and monitor the presence or absence of abnormalities in the work machine 100. The information processing device 200 can also display data representing the operating status of the work machine 100, for example, via a display device 208 (described below), allowing the user to check the data. The information processing device 200 can also, for example, train a learning model to learn the operating status of the work machine 100, and generate a trained model for supporting the operation of the work machine 100.

[0033] Furthermore, the information processing device 200 may transmit to the work machine 100 various data such as programs and reference data used in the processing of the control device 110, etc. This allows the work machine 100 to perform various processes related to the operation of the work machine 100 using the various data downloaded from the information processing device 200.

[0034] The sensor group 300 is provided at the work site of the work machine 100 .

[0035] For example, if the operation support system SYS includes multiple work machines 100, a sensor group 300 is provided for each work machine 100. Furthermore, if multiple work machines 100 included in the operation support system SYS perform work at the same work site, one sensor group 300 may be shared by the multiple work machines 100.

[0036] The sensor group 300 includes sensors 300-1 to 300-M (M: an integer equal to or greater than 2). Hereinafter, the sensors 300-1 to 300-M may be collectively referred to as sensor 300-X.

[0037] Sensors 300-1 to 300-M measure the state of objects at the work site of work machine 100 and acquire measurement data representing that state. Objects at the work site include the work target of work machine 100 (for example, earth and sand in a specified area at the work site of shovel SVL or bulk cargo M inside the hold HD of vessel SP), as well as people such as workers present at the work target and in its vicinity. Objects at the work site may also include other work machines such as shovels and bulldozers in the work target and its vicinity, and work vehicles such as trucks for transporting earth and sand. The state of an object includes the shape and characteristics of the object.

[0038] The sensors 300-1 to 300-M include, for example, ranging sensors (also referred to as "distance sensors"). Ranging sensors include, for example, LIDAR (Light Detecting and Ranging), millimeter-wave radar, ultrasonic sensors, infrared sensors, and the like. The sensors 300-1 to 300-M may also include, for example, a 3D camera capable of acquiring data representing distance (depth) in addition to two-dimensional images, such as a stereo camera or a TOF (Time Of Flight) camera. The sensors 300-1 to 300-M may also include a mixture of ranging sensors and 3D cameras. This enables the sensor group 300 to acquire measurement data representing the shape of objects at the work site around the work machine 100. Hereinafter, sensors capable of acquiring measurement data representing the shape of objects, such as ranging sensors and 3D cameras, may be referred to as "shape sensors" for convenience.

[0039] Furthermore, the sensors 300-1 to 300-M may include a multi-wavelength spectroscopic camera. Examples of multi-wavelength spectroscopic cameras include a multispectral camera and a hyperspectral camera. This allows the sensor group 300 to acquire measurement data that represents the characteristics of objects at a work site, such as the hardness and moisture content of soil and sand. Hereinafter, for convenience, sensors that can acquire measurement data that represent the characteristics of objects, such as multi-wavelength spectroscopic cameras, may be referred to as "characteristic sensors."

[0040] For example, sensors 300-1 to 300-M include multiple shape sensors. The multiple shape sensors may be installed in different locations on the work site around the work machine 100, and such that the sensing range of each sensor overlaps with the sensing range of at least one other shape sensor. This may allow, for example, even if occlusion occurs in the measurement data of one shape sensor and measurement data representing the shape of an object in part of its sensing range cannot be acquired, other shape sensors may be able to acquire measurement data representing the shape of objects in that range. This allows the sensor group 300 to more reliably acquire measurement data representing the shape of objects on the work site of the work machine 100.

[0041] Furthermore, sensors 300-1 to 300-M may include multiple characteristic sensors. The multiple characteristic sensors may be installed in different locations on the work site around work machine 100, and each sensorable range may overlap with at least one other characteristic sensor. This may allow, for example, even if occlusion occurs in the measurement data of one characteristic sensor and measurement data representing the characteristics of a portion of the sensing range cannot be obtained, other shape sensors may be able to obtain measurement data representing the characteristics of objects in that range. Therefore, sensor group 300 can more reliably obtain measurement data representing the characteristics of objects on the work site of work machine 100.

[0042] Furthermore, sensors 300-1 to 300-M may include a sensor having both the function of a shape sensor and the function of a characteristic sensor (hereinafter referred to as an "integrated sensor"). In this case, sensors 300-1 to 300-M may include a plurality of integrated sensors. The plurality of characteristic sensors may be provided at different locations in the work site around work machine 100, and each sensorable range may overlap with at least one other characteristic sensor.

[0043] Sensors 300-1 to 300-M may be fixed to the work site, or may be mounted on a mobile object that can move within the work site of work machine 100. Mobile objects include, for example, work machines and work vehicles that move within the work site. Furthermore, mobile objects that can move within the work site may include, for example, flying objects such as drones that fly over work targets and the like at the work site.

[0044] The outputs (measurement data) of sensors 300-1 to 300-M are taken into information processing device 200 via communication line NW. The outputs of sensors 300-1 to 300-M are taken into information processing device 200 directly, for example, via communication line NW. Furthermore, the outputs of sensors 300-1 to 300-M may be once taken into work machine 100 via communication line NW, and then taken into information processing device 200 via work machine 100. Furthermore, when sensors 300-1 to 300-M are mounted on a specific device such as the mobile body described above, the outputs of sensors 300-1 to 300-M may once be taken into the specific device, and then taken from that device into information processing device 200.

[0045] The sensor group 300 may simply include only one shape sensor or one characteristic sensor. Furthermore, instead of the sensor group 300, the operation support system SYS may simply include only one sensor that is capable of acquiring measurement data that represents the state of objects at the work site around the work machine 100. Furthermore, the sensor group 300 may be omitted. In this case, the measurement data that represents the state of objects at the work site may be acquired by a periphery monitoring sensor 140 mounted on the work machine 100 and transmitted to the information processing device 200.

[0046] [Hardware Configuration of Operation Support System] Next, the hardware configuration of the operation support system SYS will be described with reference to FIG. 3 in addition to FIG. 1 and FIG. 2.

[0047] The hardware configuration of the remote operation support device 400 may be the same as that of the information processing device 200. Therefore, illustration and description of the hardware configuration of the remote operation support device 400 will be omitted.

[0048] <Hardware configuration of work machine> As shown in FIG. 1 , the work machine 100 includes a control device 110, an actuator 120, a work device 125, an operation device 130, a periphery monitoring sensor 140, a motion monitoring sensor 150, an output device 160, an input device 170, and a communication device 180.

[0049] The control device 110 controls the work machine 100 .

[0050] The functions of the control device 110 are realized by any hardware or any combination of hardware and software, etc. The control device 110 includes, for example, an auxiliary storage device 110A, a memory device 110B, a CPU (Central Processing Unit) 110C, and an interface device 110D, which are connected by a bus BS1.

[0051] The auxiliary storage device 110A is a non-volatile storage means that stores the programs to be installed as well as necessary files, data, etc. The auxiliary storage device 110A is, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory) or a flash memory.

[0052] For example, when an instruction to start a program is received, the memory device 110B loads the program from the auxiliary storage device 110A so that it can be read by the CPU 110C. The memory device 110B is, for example, an SRAM (Static Random Access Memory).

[0053] The CPU 110C executes, for example, a program loaded into the memory device 110B, and realizes various functions of the control device 110 according to instructions of the program.

[0054] The interface device 110D functions, for example, as a communication interface for connecting to a communication line inside the work machine 100. The interface device 110D may include a plurality of different types of communication interfaces in accordance with the types of communication lines to be connected.

[0055] The interface device 110D also functions as an external interface for reading data from a recording medium and writing data to a recording medium. The recording medium is, for example, a dedicated tool that is connected via a detachable cable to a connector installed inside the cabin 10. The recording medium may also be a general-purpose recording medium, such as an SD memory card or a USB (Universal Serial Bus) memory. As a result, a program that realizes the various functions of the control device 110 may be provided, for example, by a portable recording medium, and installed in the auxiliary storage device 110A of the control device 110. The program may also be downloaded from another computer (for example, the information processing device 200) external to the work machine 100 via the communication device 180, and installed in the auxiliary storage device 110A.

[0056] Note that some of the functions of the control device 110 may be realized by another control device (control device). In other words, the functions of the control device 110 may be realized in a distributed manner by a plurality of control devices mounted on the work machine 100.

[0057] Actuator 120 drives the driven elements of work machine 100. Actuator 120 may be a hydraulic actuator that drives the driven elements hydraulically, or an electric actuator that drives the driven elements electrically, or may drive the driven elements using another drive method.

[0058] The working device 125 directly acts on the work target to perform a predetermined task. The working device 125 includes one or more driven elements and is driven by the actuator 120. The working device 125 is, for example, an attachment AT of the shovel SLV described below or a bucket elevator 59 of the continuous unloader ULD described below. The working device 125 also includes a working part corresponding to the tip of the working device 125 that is used to make changes to the work target. The working part is, for example, a bucket 6 of the shovel SVL described below or a scraping part 61 of the continuous unloader ULD described below.

[0059] The operating device 130 is provided in the driver's seat of the work machine 100, and is used by an operator sitting in the driver's seat to operate the various driven elements of the work machine 100. Specifically, the operating device 130 is used by the operator to operate the actuators 120 that drive the respective driven elements, and as a result, it is possible for the operator to operate the driven elements that are driven by the actuators 120. The operating device 130 includes, for example, a lever device that is operated by the operator's hand, or a pedal device that is operated by the operator's foot.

[0060] The operating device 130 is, for example, electric. In this case, the operating device 130 outputs an electric signal (hereinafter referred to as an "operation signal") corresponding to the operation content, and the operation signal is input to the control device 110. The control device 110 then controls the actuator 120 to operate according to the content of the operation signal. For example, if the actuator 120 is a hydraulic actuator such as a hydraulic cylinder or a hydraulic motor, the control device 110 outputs a control signal corresponding to the content of the operation signal to a predetermined hydraulic control valve (hereinafter referred to as an "operation control valve") (e.g., a proportional valve). As a result, a pilot pressure corresponding to the operation content of the operating device 130 is input from the operation control valve to a hydraulic control device (e.g., a control valve). Therefore, the hydraulic control device can drive the actuator 120 according to the content of the operation signal, and as a result, the actuator 120 can perform an operation corresponding to the operation content of the operating device 130. Furthermore, for example, if the actuator 120 is an electric actuator such as an electric motor (motor), the control device 110 outputs a control signal corresponding to the content of the operation signal to a drive device (e.g., a motor driver) that drives the actuator 120. This allows the drive device to drive the actuator 120 in accordance with the content of the operation signal, and as a result, the actuator 120 can perform an operation in accordance with the content of the operation on the operation device 130. Furthermore, if the actuator 120 is an electric actuator, the operation signal may be input directly to the drive device.

[0061] The operating device 130 may also be a hydraulic pilot type. In this case, the operating device 130 uses a predetermined hydraulic source (e.g., a pilot pump) to output a pilot pressure (hereinafter, "operation pilot pressure") corresponding to the operation content. For example, if the actuator 120 is a hydraulic actuator, the operation pilot pressure is input to a hydraulic control device. This allows the hydraulic control device to drive the actuator 120 according to the operation pilot pressure, and as a result, the actuator 120 can perform an operation corresponding to the operation content of the operating device 130. For example, if the actuator 120 is an electric actuator, a detected value of the operation pilot pressure is input to the control device 110, and the control device 110 outputs a control signal corresponding to the detected value of the operation pilot pressure to the drive device. This allows the drive device to drive the actuator 120 according to the operation pilot pressure, and as a result, the actuator 120 can perform an operation corresponding to the operation content of the operating device 130.

[0062] The periphery monitoring sensor 140 acquires measurement data that indicates the state of the work target of the work machine 100 and objects in its surroundings.

[0063] The perimeter monitoring sensor 140 includes, for example, a shape sensor capable of acquiring measurement data representing the shape of the work target and surrounding objects. Shape sensors include, for example, an imaging device (e.g., a monocular camera) capable of acquiring two-dimensional image data, a distance measurement sensor, a 3D camera, etc. The output of the perimeter monitoring sensor 140 is input into the control device 110. This allows the control device 110 to grasp the state of the work target of the work machine 100 and the surrounding objects. The perimeter monitoring sensor 140 may also include a characteristic sensor capable of acquiring measurement data representing the characteristics of the work target and surrounding objects.

[0064] The operation monitoring sensor 150 acquires measurement data that represents the operating state of the work machine 100. For example, the operation monitoring sensor 150 acquires measurement data that represents the operating state of the work implement 125. The output of the operation monitoring sensor 150 is taken into the control device 110. This allows the control device 110 to grasp the operating state of the work machine 100. Therefore, the control device 110 can, for example, perform control related to the automatic driving function while grasping the operating state of the work machine 100.

[0065] The output device 160 outputs various types of information to the user of the work machine 100 (for example, the operator in the driver's seat) and people in the vicinity of the work machine 100 (for example, workers or drivers of work vehicles).

[0066] For example, the output device 160 includes lighting equipment, a display device 162, etc. that output various types of information in a visual manner.

[0067] The lighting device is, for example, a warning light (indicator lamp), etc. The display device 162 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, etc. For example, the lighting device and the display device 162 may be provided inside the driver's seat and output various types of information to an operator or the like inside the driver's seat in a visual manner.

[0068] Furthermore, the lighting equipment and display device 162 may be provided so as to be exposed to the outside of the work machine 100, for example, and may output various types of information to workers and the like in the vicinity of the work machine 100 in a visual manner.

[0069] The display device 162 also includes, for example, a portable display device that can be carried by a user such as an operator. The portable display device is used by a user such as an operator by bringing it into the cabin 10 of the excavator SVL. The portable display device also includes, for example, a wearable display device that can display image information on the transparent surface of glasses or goggles that the user can wear. The wearable display device can display image information superimposed on the user's field of view, assuming that the user can see the surrounding situation through the transparent surface. Data regarding the position and orientation of the wearable display device is input to the control device 110 via a short-range communication line based on a wireless communication standard such as Bluetooth or Wi-Fi. This allows the control device 110 to grasp the viewpoint and field of view of the user (operator) wearing the wearable display device and display content on the transparent surface that matches the field of view. Examples of wearable display devices include smart glasses and AR (Augmented Reality) goggles. The portable display device also includes, for example, a display device installed in a mobile terminal carried by the user.

[0070] The output device 160 may also include a sound output device 164 that outputs various types of information auditorily. The sound output device 164 includes, for example, a buzzer, a speaker, etc. The sound output device 164 may be provided, for example, inside or outside the driver's seat, and may output various types of information auditorily to an operator inside the driver's seat or to people (workers, etc.) around the work machine 100.

[0071] The output device 160 may also include a device that outputs various information in a tactile manner, such as by vibrating the cockpit.

[0072] The input device 170 accepts various inputs from the user of the work machine 100, and signals corresponding to the accepted inputs are taken into the control device 110. For example, the input device 170 is provided inside the driver's seat and accepts inputs from an operator or the like inside the driver's seat. Alternatively, the input device 170 may be provided so as to be exposed outside the work machine 100 and accept inputs from workers or the like in the vicinity of the work machine 100.

[0073] For example, the input device 170 includes an operation input device that accepts input by mechanical operation from the user. The operation input device may include a touch panel mounted on the display device 162, a touch pad installed around the display device 162, a button switch, a lever, a toggle, a knob switch provided on the operation device 130 (lever device), and the like.

[0074] The input device 170 may also include an audio input device that accepts audio input from the user. The audio input device includes, for example, a microphone.

[0075] The input device 170 may also include a gesture input device that accepts gesture inputs from the user. The gesture input device includes, for example, an imaging device that captures an image of a gesture being made by the user.

[0076] The input device 170 may also include a biometric input device that accepts biometric input from the user, such as input of biometric information such as the user's fingerprint or iris.

[0077] The communication device 180 connects to an external communication line NW and communicates with devices provided separately from the work machine 100. Devices provided separately from the work machine 100 may include devices external to the work machine 100, as well as portable terminal devices (mobile terminals) that are brought into the driver's seat by the user of the work machine 100. The communication device 180 may be, for example, a 4G (4 th Generation) and 5G (5 th The communication device 180 may include a mobile communication module conforming to a standard such as the IEEE 802.11 Generation. The communication device 180 may also include, for example, a satellite communication module. The communication device 180 may also include, for example, a Wi-Fi communication module or a Bluetooth (registered trademark) communication module. If there are multiple connectable communication lines NW, the communication device 180 may include multiple communication devices according to the types of the communication lines NW.

[0078] For example, the communication device 180 communicates with external devices such as the information processing device 200 and the remote operation support device 400 at the work site through a local communication line established at the work site. The local communication line is, for example, a local 5G (so-called local 5G) mobile communication line established at the work site or a local network using Wi-Fi 6.

[0079] The communication device 180 may also communicate with the information processing device 200 and the remote operation support device 400 outside the work site via a wide area communication line including the work site, that is, a wide area network.

[0080] <Hardware Configuration of Information Processing Apparatus> FIG. 3 is a block diagram showing an example of the hardware configuration of the information processing apparatus 200. As shown in FIG.

[0081] The functions of the information processing device 200 are realized by any hardware or any combination of hardware and software, etc. For example, as shown in Fig. 5, the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a high-speed calculation device 205, a communication interface 206, an input device 207, a display device 208, and a sound output device 209. These are connected by a bus BS2.

[0082] The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. Examples of the recording medium 201A include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, etc. This allows the information processing device 200 to read various data used in processing through the recording medium 201A, store the data in the auxiliary storage device 202, and install programs that realize various functions.

[0083] The information processing device 200 may acquire various data and programs used in processing from an external device via the communication interface 206 .

[0084] The auxiliary storage device 202 stores various installed programs, as well as files and data necessary for various processes. The auxiliary storage device 202 includes, for example, a hard disk drive (HDD), a solid state disk (SSD), a flash memory, etc.

[0085] When an instruction to start a program is received, the memory device 203 reads and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, a dynamic random access memory (DRAM) or an SRAM.

[0086] The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and realizes various functions related to the information processing device 200 in accordance with the programs.

[0087] The high-speed arithmetic unit 205 performs arithmetic processing at a relatively high speed in cooperation with the CPU 204. The high-speed arithmetic unit 205 includes, for example, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0088] The high-speed calculation device 205 may be omitted depending on the required calculation processing speed.

[0089] The communication interface 206 is used as an interface for connecting to an external device so that communication is possible. This allows the information processing device 200 to communicate with external devices such as the work machine 100 through the communication interface 206. Furthermore, the communication interface 206 may have multiple types of communication interfaces depending on the communication method between the connected devices, etc.

[0090] The input device 207 receives various inputs from a user. The input device 207 includes a remote control operation device for remotely operating the work machine 100.

[0091] The input device 207 includes, for example, an input device that accepts mechanical operation input from a user (hereinafter referred to as an "operation input device"). The operation device for remote operation may be an operation input device. The operation input device includes, for example, a button, a toggle, a lever, a keyboard, a mouse, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, etc.

[0092] The input device 207 may also include a voice input device capable of receiving voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

[0093] The input device 207 may also include a gesture input device capable of receiving gesture input from a user. The gesture input device may include, for example, a camera capable of capturing images of the user's gestures.

[0094] The input device 207 may also include a biometric input device capable of accepting biometric input from a user. The biometric input device may include, for example, a camera capable of acquiring image data containing information representing a user's fingerprint or iris.

[0095] The display device 208 displays an information screen or an operation screen for the user of the information processing device 200. The display device 208 is, for example, a liquid crystal display or an organic EL display.

[0096] The sound output device 209 conveys various pieces of information by sound to the user of the information processing device 200. The sound output device 209 is, for example, a buzzer, an alarm, a speaker, or the like.

[0097] [First Example of Functional Configuration of Operation Support System] Next, a first example of the functional configuration of the operation support system SYS will be described with reference to FIGS. 4 and 5 in addition to FIGS. 1 to 3.

[0098] Fig. 4 is a functional block diagram showing a first example of the functional configuration of the operation support system SYS. Fig. 5 is a diagram for explaining an example of a method for calculating the safety level.

[0099] Hereinafter, the term "trajectory of the working portion of the work machine 100" is used to include both the path (i.e., trajectory) that the working portion of the work machine 100 has already traveled, and the path that it may travel in the future.

[0100] In this example, the operation support system SYS provides support to a user who operates the work machine 100 and performs work.

[0101] As shown in FIG. 4, the control device 110 of the work machine 100 includes, as functional units, an operation log providing unit 1101 and a work support unit 1102.

[0102] Furthermore, when the operation support system SYS includes a plurality of work machines 100, there may be work machines 100 in which the control device 110 includes only the operation log providing unit 1101 and the work support unit 1102, and work machines 100 in which the control device 110 includes only the latter. In this case, the former work machine 100 only has the function of acquiring the operation log of the work machine 100 and providing it to the information processing device 200, which is used for the work support function of the latter work machine 100. The same may be true for the second example of the operation support system SYS described below.

[0103] The information processing device 200 includes, as functional units, a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a teacher data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007.

[0104] The operation log providing unit 1101 is a functional unit for acquiring an operation log during a predetermined operation of the work machine 100 and providing it to the information processing device 200 .

[0105] The predetermined operation of the work machine 100 includes, for example, an excavation operation, a boom-raising and swinging operation, a boom-lowering and swinging operation, an earth-discharging operation, a broom operation, etc., which are used during excavation work by the shovel SVL. The predetermined operation of the work machine 100 may also include an excavation operation, an earth-discharging operation, a sweeping operation, a leveling operation, a rolling operation, a broom operation, etc., which are used during ground leveling work by the shovel SVL. The predetermined operation of the work machine 100 may also include a cutting operation, a rolling operation, etc., which are used during slope work by the shovel SVL. The predetermined operation of the work machine 100 is, for example, a scraping operation of loose loads M, which is used during unloading work by the continuous unloader ULD.

[0106] The sweeping operation is, for example, an operation in which the attachment AT is operated to push the bucket 6 forward along the ground, thereby sweeping soil and sand forward with the back surface of the bucket 6. In the sweeping operation, for example, the attachment AT lowers the boom 4 and opens the arm 5. The horizontal towing operation is, for example, an operation in which the attachment AT is operated to move the tip of the bucket 6 approximately horizontally along the ground, pulling it toward the user, thereby leveling out unevenness in the ground (surface of the terrain). In the horizontal towing operation, for example, the attachment AT raises the boom 4 and closes the arm 5. The compaction operation is, for example, an operation in which the attachment AT is operated to press the ground with the back surface of the bucket 6. Furthermore, the compaction operation may also be an operation in which the bucket 6 is moved up and down, while pressing the back surface of the bucket 6 against the ground. The compaction operation may also be an operation in which the bucket 6 is pushed forward along the ground, the back surface of the bucket 6 sweeps soil and sand to a predetermined position forward, and then the back surface of the bucket 6 presses the ground at the predetermined position against the ground. In the compaction operation, for example, the attachment AT lowers the boom 4 while pressing the ground. The broom operation may be, for example, an operation in which the upper rotating body 3 is operated to rotate the bucket 6 left and right while keeping it along the ground. In the broom operation, for example, the attachment AT and the upper rotating body 3 are operated to push the bucket 6 forward while rotating the bucket 6 alternately left and right while keeping it along the ground. In the broom operation, for example, the upper rotating body 3 alternately rotates left and right. In the broom operation, for example, in addition to the alternate left and right rotation of the upper rotating body 3, the attachment AT may lower the boom 4 and open the arm 5, as in the sweeping operation.

[0107] The operation log of the work machine 100 is time-series data that represents the operating state of the work machine 100. For example, the operation log of the work machine 100 includes time-series data that represents the details of the operation by the operator. The time-series data that represents the details of the operation by the operator is, for example, time-series measurement data of the operating pilot pressure output from the hydraulic pilot type operating device 130 or time-series output data of the electric operating device 130 (i.e., time-series data of the operation signal). The operation log of the work machine 100 may also be time-series output data of the operation monitoring sensor 150.

[0108] For example, the operation log provider 1101 acquires an operation log from when the work machine 100 is operated by an operator with a long history of operating the work machine 100 and a relatively high level of experience (hereinafter referred to as an "expert" for convenience), and provides this to the information processing device 200. As a result, as will be described later, it is possible to generate a trained model (for example, trained model LM3, described later) that can reproduce the operation of the work machine 100 operated by an expert, by machine learning based on the operation log of the work machine 100. Furthermore, the operation log of the work machine 100 provided to the information processing device 200 may include an operation log from when an operator other than the expert operated the work machine 100. Furthermore, the operation log of the work machine 100 may be acquired separately in accordance with the type of trained model generated by the information processing device 200.

[0109] The operation log providing unit 1101 includes an operation log recording unit 1101A, an operation log storage unit 1101B, and an operation log transmission unit 1101C.

[0110] The operation log recording unit 1101A acquires an operation log during a predetermined operation of the work machine 100 and records it in the operation log storage unit 1101B. For example, each time a predetermined operation of the work machine 100 is executed, the operation log recording unit 1101A records an operation log during that operation in the operation log storage unit 1101B.

[0111] The operation log storage unit 1101B stores the operation log of the work machine 100. For example, the operation log storage unit 1101B stores, for each predetermined operation performed by the work machine 100, an operation log and data on the time (for example, year and date) when the predetermined operation was performed, linked together. The data on the time when the predetermined operation was performed includes data on both the start and end times of the predetermined operation of the work machine 100. Furthermore, when multiple predetermined operations are defined, the operation log storage unit 1101B stores, for each predetermined operation performed by the work machine 100, an operation log, data on the time when the predetermined operation was performed, and data on identification information for the performed predetermined operation, linked together. Hereinafter, data linked to the operation log of the work machine 100 may be referred to as "associated data" for convenience. For example, the operation log storage unit 1101B accumulates record data that indicates the correspondence between the operation log and associated data for each predetermined operation performed by the work machine 100, thereby building a database of operation logs of the work machine 100 when the predetermined operations are performed.

[0112] The operation log stored in the operation log storage unit 1101B that has already been transmitted to the information processing device 200 by an operation log transmission unit 1101C (described later) may be deleted afterward.

[0113] The operation log transmission unit 1101C transmits the operation logs stored in the operation log storage unit 1101B when the work machine 100 performs a predetermined operation, and the associated data linked to the operation logs, to the information processing device 200 via the communication device 180. The operation log transmission unit 1101C may also transmit record data indicating the correspondence between the operation logs of the work machine 100 and the associated data for each predetermined operation performed by the work machine 100 to the information processing device 200.

[0114] For example, the operation log transmission unit 1101C transmits the untransmitted operation log and associated data of the work machine 100 stored in the operation log storage unit 1101B to the information processing device 200 in response to a request to transmit the operation log of the work machine 100 received from the information processing device 200. Furthermore, the operation log transmission unit 1101C may automatically transmit the untransmitted operation log and associated data of the work machine 100 stored in the operation log storage unit 1101B to the information processing device 200 at a predetermined timing. The predetermined timing is, for example, when the work machine 100 stops operating (key switch is turned off) or starts operating (key switch is turned on), etc.

[0115] The log acquisition unit 2001 acquires a log when the work machine 100 executes a predetermined operation.

[0116] The log when the work machine 100 performs a predetermined operation includes an operation log when the work machine 100 performed the predetermined operation and a status log of the work object. The status log of the work object includes data that represents the status of the work object before and after the work machine 100 performs the predetermined operation. The status of the work object includes the shape of the work object, the characteristics of the work object, and the like. The operation log when the work machine 100 performs a predetermined operation is uploaded from the work machine 100 based on the output of the operation monitoring sensor 150. The status log of the work object when the work machine 100 performs a predetermined operation is acquired based on the measurement data uploaded from the sensor group 300 and the associated data uploaded from the work machine 100 (data of the time the predetermined operation was performed). Furthermore, the status log of the work object when the work machine 100 performs a predetermined operation may be uploaded from the work machine 100 based on the output of the perimeter monitoring sensor 140.

[0117] The simulator section 2002 performs a computer simulation of a predetermined operation of the work machine 100 using a virtual model of the work machine 100 and the work target.

[0118] For example, the distinct element method (DEM) is used to model the soil and bulk cargo M to be worked on as a collection of minute particles. As a result, the simulator unit 2002 can virtually reproduce the overall behavior of the soil and bulk cargo M to be worked on as a collection by having the virtual model of the work machine 100 perform a predetermined operation such as an excavation operation or a scraping operation and analyzing the movement of each minute particle.

[0119] The simulator section 2002 acquires data representing the operating state of the work machine 100 and data on the state of the work object before and after the execution of the predetermined operation as logs when the work machine 100 executes a predetermined operation through computer simulation. The former data corresponds to the operation log when the work machine 100 executes a predetermined operation through computer simulation, and the latter data corresponds to the state log of the work object when the work machine 100 executes a predetermined operation through computer simulation.

[0120] The simulator unit 2002 performs computer simulations of numerous patterns relating to the predetermined operation of the work machine 100, using various conditions of the work object (earth and sand) and various trajectories of the working parts of the work machine 100. In this way, the simulator unit 2002 can accumulate in the log storage unit 2003 logs of when the work machine 100 performs the predetermined operation through computer simulation under mutually different conditions.

[0121] The log storage unit 2003 stores in an accumulated form logs acquired by the log acquisition unit 2001 and the simulator unit 2002 when the work machine 100 performs a predetermined operation. For example, the log storage unit 2003 stores an operation log for each predetermined operation that the work machine 100 actually performs or performs by computer simulation, a status log of the work target, and associated data in a linked form. In the log storage unit 2003, the logs acquired by the log acquisition unit 2001 and the logs acquired by the simulator unit 2002 may be stored in a distinguishable manner, or may be stored mixed together in an indistinguishable manner.

[0122] The teacher data generation unit 2004 generates teacher data for machine learning based on the logs of when the work machine 100 performs a predetermined operation, which are stored in the log storage unit 2003, and outputs a teacher data set that is a collection of a large number of pieces of teacher data. The teacher data generation unit 2004 may generate teacher data automatically by batch processing, or may generate teacher data in response to input from the user of the information processing device 200. The teacher data generation unit 2004 includes teacher data generation units 2004A and 2004B.

[0123] The teacher data generation unit 2004A generates a teacher data set TRD1 for generating a trained model LM1.

[0124] The learned model LM1 uses specified data as input to infer a predicted trajectory of the working part due to the operation of the operator. The specified data input to the learned model LM1 includes, for example, data representing the current operation state of the work machine 100 by the operator. This allows the learned model LM1 to predict the trajectory of the working part immediately after the current operation state of the work machine 100 by the operator. The data representing the operation state of the work machine 100 is, for example, data on the operation state itself of the work machine 100, that is, data representing the operation input from the operator. Furthermore, the data representing the operation state of the work machine 100 may be data representing the operating state of the work machine 100 that reflects the operation state of the work machine 100. Furthermore, the specified data input to the learned model LM1 may be a history of data representing the operation state of the work machine 100, including data representing the current operation state of the work machine 100 by the operator. This allows the learned model LM1 to more accurately predict the trajectory of the working part immediately after the operation of the work machine 100 from the history of the most recent operation state of the work machine 100 by the operator. Furthermore, the predetermined data input to the trained model LM1 may include data representing the shape of the work object. This enables the trained model LM1 to predict the trajectory of the immediate work part that will be executed by the operator's operation, starting from the current shape of the work object. The predetermined data input to the trained model LM1 may include only one of data representing the operation state of the work machine 100 by the operator and data representing the shape of the work object, or may include both.

[0125] Each training data included in the training data set TRD1 is a combination of the above-mentioned specified data as input data and data representing the trajectory (track) of the work area executed based on the input data as correct output data.

[0126] The teacher dataset TRD1 for generating the trained model LM1 may be generated from only the logs acquired by the log acquisition unit 2001 and the logs output from the simulator unit 2002. In this case, the simulator unit 2002 may be omitted. Similarly, the teacher dataset TRD1 for generating the trained model LM1 may be generated from only the logs acquired by the log acquisition unit 2001 and the logs output from the simulator unit 2002. In this case, the operation log providing unit 1101 of the sensor group 300 and the work machine 100 may be omitted. Furthermore, the teacher dataset TRD1 for generating the trained model LM1 may include a base teacher dataset and a teacher dataset for final adjustment (fine tuning). In this case, since a large amount of data is required, the base teacher dataset may be generated based on the logs output from the simulator unit 2002, and the teacher dataset for final adjustment may be generated based on the logs acquired by the log acquisition unit 2001. The same may be true for the training data set TRD2 for generating the trained model LM2 described below and the training data set TRD3 for generating the trained model LM3.

[0127] The teacher data generation unit 2004B generates a teacher data set TRD2 for generating a trained model LM2.

[0128] The trained model LM2 uses predetermined data as input to infer the shape of the work object after the work machine 100 has performed a predetermined operation. "After the work machine 100 has performed a predetermined operation" refers to "after one predetermined operation has been performed" in a repeated pattern, such as the excavation operation of a shovel SVL, in which one predetermined operation is performed, followed by a time interval before the next predetermined operation is performed. Furthermore, "after the work machine 100 has performed a predetermined operation" refers to "after the predetermined operation has been performed a predetermined time from the start point" in a pattern in which predetermined operations are performed continuously, such as the scraping operation of a continuous unloader ULD. The predetermined data input to the trained model LM2 includes, for example, data representing the prior shape of the work object or the work object and its surrounding objects (hereinafter referred to as "work object, etc."), and data on the trajectory (i.e., track) of the work part when the work machine 100 performs a predetermined operation. This allows the trained model LM2 to predict the changed shape of the work object, etc., from the track of the work part of the work machine 100, based on the prior shape of the work object, etc. The time when a predetermined operation of the work machine 100 is being performed refers to the time when the predetermined operation is performed once if the predetermined operation is repeated at intervals. Also, the time when a predetermined operation of the work machine 100 is being performed refers to the time when the predetermined operation is being performed a predetermined time from the start point if the predetermined operation is performed continuously.

[0129] Furthermore, the input data of the trained model LM2 may include data representing the characteristics of the work object of the work machine 100. For example, the data representing the characteristics of the work object includes data on the angle of repose of the soil or bulk cargo M of the work object. This allows the trained model LM2 to infer a more appropriate shape of the work object by taking into account the angle of repose of the soil or bulk cargo M. In this case, the input data in the training data includes data representing the characteristics of the work object (angle of repose data).

[0130] Furthermore, the predetermined data input to the trained model LM2 may include data representing the reaction force from the work target to the working part when a predetermined operation is performed by the work machine 100. As a result, the trained model LM2 can estimate an appropriate shape of the soil by taking into account the reaction force of the working part (bucket 6), even in cases where, for example, the bucket 6 of the shovel SVL hits a rock or the like underground and, as a result, the soil cannot be excavated.

[0131] The predetermined data input to the trained model LM2 may also include data representing occlusion corresponding to data representing the shape of the work object, etc., prior to the event. Occlusion refers to a state in which observation is impossible due to the presence of an obstruction or the like between the perimeter monitoring sensor 140 or sensor 300-X and the work object. The data representing occlusion is, for example, a collection of information indicating whether each small area that divides the observation target area is observable. As a result, for example, when occlusion is included in the data representing the shape of the work object, etc., prior to the event, the trained model LM2 can predict the shape of the work object, etc., in the future by interpolating the occluded area from data representing the surrounding shape. In this case, the trained model LM2 performs machine learning using training data that intentionally includes occlusion.

[0132] Each piece of training data included in the training data set TRD2 is a combination of the above-mentioned predetermined data as input data and the shape of the work object after the work machine 100 has performed a predetermined operation as correct output data.

[0133] The machine learning unit 2005 generates trained models LM1 and LM2 by performing machine learning on the base learning model based on the teacher data set generated by the teacher data generation unit 2004. The trained model (base learning model) includes, for example, a neural network such as a deep neural network (DNN).

[0134] The machine learning unit 2005 includes machine learning units 2005A and 2005B.

[0135] The machine learning unit 2005A causes the base learning model M1 to perform machine learning based on the teacher data set TRD1 output from the teacher data generation unit 2004A. As a result, the machine learning unit 2005A can generate a learned model LM1 that is capable of outputting (inferring) data representing a predicted trajectory of a working part of the work machine 100, using data related to the operation state of the work machine 100 and data representing the shape of the work target as input. The machine learning unit 2005A may also correct (additionally learn) the learned model LM1 so as to reduce the error between the inference result from the learned model LM1 and the actual measurement result of the operation monitoring sensor 150. In this case, the inference result from the learned model LM1 and the data of the actual measurement result of the operation monitoring sensor 150 are uploaded from the work machine 100 to the information processing device 200.

[0136] The machine learning unit 2005B performs machine learning on the base learning model M2 based on the teacher data set TRD2 output from the teacher data generation unit 2004B. As a result, the machine learning unit 2005B can generate a learned model LM2 that can output (infer) data representing the shape of the work object after the work machine 100 has performed a predetermined operation, using inputs such as data representing the shape of the work object in advance and data representing the trajectory of the work part when the work machine 100 performs a predetermined operation. The machine learning unit 2005B may also correct (additionally learn) the learned model LM2 so as to reduce the error between the inference results from the learned model LM2 and the actual measurement results from the perimeter monitoring sensor 140 and the sensor group 300. In this case, the inference results from the learned model LM2 and the actual measurement result data from the perimeter monitoring sensor 140 are uploaded from the work machine 100 to the information processing device 200. The actual measurement result data from the sensor group 300 is uploaded from the sensor group 300 to the information processing device 200.

[0137] The trained model storage unit 2006 stores trained models LM1 and LM2 output by the machine learning unit 2005. Furthermore, when the machine learning unit 2005A re-learns or additionally trains the trained model LM1, the trained model LM1 in the trained model storage unit 2006 is updated. The same applies when the trained model LM2 is re-learned or additionally trained by the machine learning unit 2005B.

[0138] The distribution unit 2007 distributes data of the learned models LM1 and LM2 to the work machine 100.

[0139] For example, when the machine learning unit 2005A generates or updates the learned model LM1, the distribution unit 2007 distributes the most recently generated or updated learned model LM1 to the work machine 100. Furthermore, in response to a signal received from the work machine 100 requesting distribution of the learned model LM1, the distribution unit 2007 may distribute the latest learned model LM1 in the learned model storage unit 2006 to the work machine 100. The same may be true for the learned model LM2.

[0140] The work support unit 1102 is a functional unit for operating the work machine 100 to carry out work, or for providing support to a user who monitors work on the work machine 100 .

[0141] The work support unit 1102 includes a learned model memory unit 1102A, an object detection unit 1102B, a safety control unit 1102C, a trajectory prediction unit 1102D, a work object shape prediction unit 1102E, a safety level calculation unit 1102F, and a safety control unit 1102G.

[0142] In addition, some or all of the functions of the object detection unit 1102B, safety control unit 1102C, trajectory prediction unit 1102D, work object shape prediction unit 1102E, safety level calculation unit 1102F, and safety control unit 1102G may be transferred to an outside of the work machine 100 (for example, to an information processing device 200).

[0143] The trained model storage unit 1102A stores trained models LM1 and LM2 that are distributed from the information processing device 200 and received via the communication device 180.

[0144] The object detection unit 1102B detects objects to be monitored in the periphery of the work machine 100 based on the output of the periphery monitoring sensor 140. Furthermore, the object detection unit 1102B may detect objects to be monitored in the periphery of the work machine 100 based on the output of a sensor group 300 arranged at the work site, in addition to the output of the periphery monitoring sensor 140. The output of the sensor group 300 may be received directly from the sensor group 300 via the communication device 180, or may be received indirectly from the sensor group 300 via the information processing device 200.

[0145] The periphery of the work machine 100 includes the periphery of the main body of the work machine 100 and the periphery of the work implement 125. The main body of the work machine 100 corresponds to, for example, the part including the undercarriage 1 and upper rotating body 3 of the excavator SVL. Furthermore, the periphery of the work machine 100 may include not only the area of ​​the work target of the work machine 100, but also the area surrounding the work target.

[0146] Objects to be monitored include, for example, people such as workers. Objects to be monitored may also include other obstacles that are in or around the work object. Other obstacles include, for example, specific moving objects in the work site of the shovel SVL, such as other work machines or work vehicles. Other obstacles may also include specific stationary objects in the work site of the shovel SVL, such as utility poles, fences, and traffic cones. Other obstacles may also include specific topographical shapes in the work site of the shovel SVL, such as ditches and holes. Other obstacles may also include, for example, wall portions that correspond to the outer edges of the hold HD.

[0147] The object detection unit 1102B performs processing to detect monitored objects in the vicinity of the work machine 100, for example, at each predetermined processing cycle, and outputs the processing results. The processing results include, for example, data indicating whether or not a monitored object has been detected, and, if a monitored object has been detected, the position of that object.

[0148] The safety control unit 1102C performs control relating to functional safety based on the output of the object detection unit 1102B.

[0149] The safety control section 1102C activates the safety function when, for example, the object detection section 1102B detects an object to be monitored within a predetermined range around the work machine 100.

[0150] The safety functions may include, for example, a notification function that outputs an alarm or the like to at least one of the inside of the driver's seat of the work machine 100, the outside of the driver's seat, and a remote operator of the work machine 100, to notify them of the detection of a monitored object. This makes it possible to alert the operator inside the driver's seat, workers around the work machine 100, and the operator remotely operating the work machine 100 that a monitored object is present in the monitoring area around the work machine 100. Below, these functions may be distinguished from one another by being referred to as an "internal notification function," a notification function for an operator inside the driver's seat, etc., a "external notification function," and a notification function for an operator remotely operating the work machine 100 or a monitor remotely monitoring the work machine 100.

[0151] Furthermore, the safety functions may include, for example, an operation restriction function that restricts the operation of the work machine 100 in response to operation of the operating device 130 or remote operation. This makes it possible to forcibly restrict the operation of the work machine 100 and reduce the possibility of the work machine 100 approaching or coming into contact with an object being monitored. The operation restriction function may include, for example, an operation deceleration function that slows down the operation speed of the work machine 100 in response to operation of the operating device 130 or remote operation compared to normal. The operation restriction function may also include an operation stop function that stops the operation of the work machine 100 and maintains the stopped state regardless of whether the operating device 130 is operated or remotely operated. The safety control unit 1102C may execute only one of the operation deceleration function and the operation stop function as the operation restriction function, or may execute both.

[0152] The safety control unit 1102C activates the alarm function, for example, when the object detection unit 1102B detects a monitored object within a predetermined range (hereinafter referred to as the "alarm range") around the work machine 100. The alarm range is, for example, a range in which the distance D from a predetermined part of the work machine 100 is equal to or less than a threshold value Dth1. The predetermined part of the work machine 100 is, for example, the main body of the work machine 100 (for example, the machine body including the undercarriage 1 and upper rotating body 3 of an excavator SVL). The predetermined part of the work machine 100 may also be a working part at the tip of the work implement 125 (for example, the bucket 6 of an excavator SVL or the scraping part 61 of a continuous unloader ULD). The threshold value Dth1 may be constant regardless of the direction as viewed from the predetermined part of the work machine 100, or may change depending on the direction as viewed from the predetermined part of the work machine 100.

[0153] The safety control unit 1102C activates an internal notification function or an external notification function using sound (i.e., an auditory method) to at least one of the inside and outside of the driver's seat, for example, by controlling the sound output device 164. At this time, the safety control unit 1102C may vary the pitch, sound pressure, tone, blowing cycle when blowing a sound periodically, sound content, etc. of the output sound according to various conditions.

[0154] The safety control unit 1102C also activates an internal notification function using, for example, a visual method. Specifically, the safety control unit 1102C may control the display device 162 inside the driver's seat to display an image indicating that a monitored object has been detected on the display device 162 together with a captured image or processed image from an imaging device included in the perimeter monitoring sensor 140. The safety control unit 1102C may also highlight the monitored object shown in the captured image or processed image displayed on the display device 162 inside the driver's seat, or a position on the image corresponding to the detected monitored object. For example, the safety control unit 1102C may superimpose a frame surrounding the detected monitored object on the image displayed on the display device 162 inside the driver's seat, or superimpose a marker on the image corresponding to the detected monitored object. This allows the control device 110 to realize a visual notification function for the operator. Furthermore, the safety control unit 1102C may use a warning light or lighting device inside the driver's seat to notify an operator or the like inside the driver's seat that an object to be monitored has been detected.

[0155] The safety control unit 1102C may also activate the external alarm function by a visual method, for example, by controlling an output device 160 (for example, a lighting device such as a headlight or a display device 162) provided on the side or the like of the work machine 100. The safety control unit 1102C may also activate the external alarm function by, for example, transmitting a command signal indicating activation of the alarm function to a terminal device (mobile terminal) carried by a person in the vicinity of the work machine 100, such as a worker, supervisor, or manager at the work site. The terminal device carried by the worker, supervisor, manager, or the like at the work site may be, for example, a general-purpose mobile terminal such as a smartphone or tablet terminal. The terminal device carried by the worker, supervisor, manager, or the like at the work site may also be a wearable terminal. The wearable terminal may be, for example, smart glasses. The safety control unit 1102C may also activate the internal alarm function by a tactile method, for example, by controlling a vibration generating device that vibrates the cockpit in which the operator sits. This enables the control device 110 to make the operator, workers and supervisors in the vicinity of the work machine 100 aware that an object to be monitored (for example, a person such as a worker) is present in a location relatively close to the periphery of the work machine 100. As a result, the control device 110 can urge the operator to check the safety conditions around the work machine 100, and urge workers in the monitored area to evacuate from the monitored area.

[0156] Furthermore, the safety control unit 1102C may activate the remote notification function by, for example, transmitting a command signal indicating activation of the notification function to the remote operation support device 400 via the communication device 180. In this case, when the remote operation support device 400 receives a command signal from the work machine 100, it may output an alarm by a visual or auditory method. This allows the operator remotely operating the work machine 100 to know that a monitored object has entered the notification range around the work machine 100 through the remote notification function via the remote operation support device 400.

[0157] The remote notification function of the safety control unit 1102C may be transferred to the remote operation support device 400. In this case, the remote operation support device 400 receives information relating to the processing results by the object detection unit 1102B from the work machine 100. Then, based on the received information, the remote operation support device 400 determines whether or not a monitored object has entered the notification range, and activates the remote notification function if a monitored object is present within the notification range.

[0158] In addition, the safety control unit 1102C may vary the notification mode (i.e., the method of notification) depending on the positional relationship between the monitored object detected within the notification range and a specified part of the work machine 100 that serves as the basis for the notification range.

[0159] For example, if a monitored object detected within the notification range by the object detection unit 1102B is located relatively far from a predetermined portion of the work machine 100, the safety control unit 1102C may output a relatively low-urgency alarm (hereinafter referred to as a "caution-level alarm") that calls attention to the monitored object. Hereinafter, for convenience, a range within the notification range that is relatively far from the predetermined portion of the work machine 100, i.e., a range that corresponds to a caution-level alarm, may be referred to as the "caution notification range." On the other hand, if a monitored object detected within the notification range by the object detection unit 1102B is located relatively close to the predetermined portion of the work machine 100, the safety control unit 1102C may output a relatively high-urgency alarm (hereinafter referred to as an "alert-level alarm") that notifies that the monitored object is approaching the predetermined portion of the work machine 100 and the risk is increasing. Hereinafter, a range within the notification range that is relatively close to the predetermined portion of the work machine 100, i.e., a range that corresponds to an alert-level alarm, may be referred to as the "alert notification range."

[0160] In this case, the safety control unit 1102C may change the pitch, sound pressure, tone, sound cycle, etc. of the sound output from the sound output device 164 and the remote operation support device 400 between a caution level alarm and an alert level alarm. The safety control unit 1102C may also change the color, shape, size, whether or not to flash, the flashing cycle, etc. of the image displayed on the display device 162 and the remote operation support device 400 indicating that a monitored object has been detected between a caution level alarm and an alert level alarm. Similarly, the safety control unit 1102C may change the color, shape, size, whether or not to flash, the flashing cycle, etc. of the monitored object or an image (e.g., a frame, marker, etc.) that highlights the position of the monitored object on the captured image or processed image of the imaging device, which is displayed on the display device 162. This allows the control device 110 to allow the operator, etc., to grasp the level of urgency, in other words, the proximity of the monitored object to a predetermined portion of the work machine 100, based on the difference in the alert sound output from the sound output device 164 and the alert image displayed on the display device 162. Furthermore, the safety control unit 1102C may issue a warning level alarm by displaying it on the display device 162 or the remote support device, and issue a warning level alarm by audio output from the sound output device 164 or the remote operation support device 400 instead of or in addition to a display. In this way, the control device 110 can allow the operator or the like to understand the level of urgency by switching the notification means.

[0161] In addition, the safety control unit 1102C may change the notification mode in multiple stages of three or more, or continuously, depending on the distance between the monitored object detected within the notification range and a specified part of the work machine 100.

[0162] For example, after the alarm function starts operating, if the monitored object that was detected by the object detection unit 1102B is no longer detected within the alarm range, the safety control unit 1102C stops the alarm function. Furthermore, after the alarm function starts operating, the safety control unit 1102C may stop the alarm function if a predetermined input for canceling the operation of the alarm function is received via the input device 170.

[0163] Furthermore, the safety control unit 1102C activates the operation restriction function, for example, when the object detection unit 1102B detects a monitored object within a predetermined range around the work machine 100 (hereinafter referred to as the "operation restriction range"). The operation restriction range may be set to be the same as the above-mentioned notification range, or may be set to be different. For example, the operation restriction range may be set to a range whose outer edge is relatively closer to a predetermined part of the work machine 100 than the notification range. As a result, the safety control unit 1102C can first activate the notification function when the monitored object enters the notification range from the outside, and then further activate the operation restriction function when the monitored object enters the inner operation restriction range. Therefore, the control device 110 can activate the notification function and the operation restriction function in stages as the monitored object moves inward within the monitoring area.

[0164] Specifically, the safety control section 1102C may activate the operation restriction function when a monitored object is detected within an operation restriction range where the distance D from a predetermined part of the work machine 100 is within a threshold value Dth2 (≦Dth1). The threshold value Dth2 may be constant regardless of the direction as viewed from the predetermined part of the work machine 100, or may change depending on the direction as viewed from the predetermined part of the work machine 100.

[0165] The operational limit range includes an operational deceleration range in which the operational speed of the work machine 100 in response to operation of the operating device 130 or remote operation is slower than normal. The operational limit range may also include an operational stop range in which the operation of the work machine 100 is stopped and maintained in a stopped state, regardless of whether or not the operating device 130 is operated or remotely operated. The operational limit range may include only one of the operational deceleration range and the operational stop range, or may include both. For example, when the operational limit range includes both the operational deceleration range and the operational stop range, the operational stop range is a range within the operational limit range that is close to a specified part of the work machine 100. The operational deceleration range is a range within the operational limit range that is set outside the operational stop range.

[0166] If the actuator 120 is a hydraulic actuator, the safety control unit 1102C activates an operation limiting function that limits the operation of the work machine 100, for example, by controlling an operation control valve. In this case, the safety control unit 1102C may limit the operation of all driven elements (i.e., the corresponding hydraulic actuators), or may limit the operation of some of the driven elements (hydraulic actuators). This allows the control device 110 to slow down or stop the operation of the work machine 100 when a monitored object is present in the vicinity of the work machine 100. Therefore, the control device 110 can prevent the work machine 100 from coming into contact with a monitored object in the vicinity of the work machine 100. The safety control unit 1102C may also activate the operation stop function by controlling a solenoid switching valve (not shown) that is located most upstream of a pilot line between a hydraulic source (e.g., a pilot pump) and an operation control valve, and blocking the pilot line. Furthermore, if the actuator 120 is an electric actuator, the safety control unit 1102C controls, for example, the drive device to decelerate or stop the actuator 120, thereby activating the operation restriction function.

[0167] Furthermore, after the operation of the operation restriction function has started, the safety control unit 1102C may stop the operation restriction function if the monitored object that was detected by the object detection unit 1102B is no longer detected within the operation restriction range. Furthermore, after the operation of the operation restriction function has started, the safety control unit 1102C may stop the operation restriction function if a predetermined input for deactivating the operation of the operation restriction function is received through the input device 170. The content of the input to the input device 170 for deactivating the alarm function and the content of the input for deactivating the operation of the operation restriction function may be the same or different.

[0168] In addition, the safety control unit 1102C may be capable of switching its function between ON (enabled) and OFF (disabled) in response to a predetermined input by an operator or the like to the input device 170 or the remote operation support device 400.

[0169] The safety control unit 1102C may be omitted.

[0170] The trajectory prediction unit 1102D predicts the trajectory of the working part of the working implement 125 in the immediate future as a result of the operator's operation, based on the operation state of the work machine 100 by the operator and the current shape of the work object. The immediate future corresponds to the period for which the trajectory of the working part is predicted, and specifically means a relatively short period in the future based on the present. The immediate future is, for example, the period from when the excavator SVL is currently performing a predetermined operation, or the period until the predetermined operation is completed assuming that it will start immediately thereafter. The immediate future may also be a period on the order of several seconds to several tens of seconds from the present. Specifically, the trajectory prediction unit 1102D acquires data representing the predicted trajectory of the working part of the working implement 125 based on data representing the operation state of the work machine 100 and data representing the current shape of the work object. The data representing the operation state of the work machine 100 is, for example, data on an operation signal corresponding to the output of the electric operation device 130, or data on the detected value of the operation pilot pressure of the hydraulic pilot operation device 130. Furthermore, data representing the current shape of the work object is acquired, for example, based on output data from the perimeter monitoring sensor 140. Furthermore, data representing the current shape of the work object may be acquired based on output data from the sensor group 300 received via the communication device 180 instead of, or in addition to, the output data from the perimeter monitoring sensor 140. Furthermore, if there is a location where the shape of the work object cannot be observed due to occlusion in the output data from the perimeter monitoring sensor 140 or the output data from the sensor group 300, data representing the shape of that location may be supplemented using a method similar to that of the work object shape prediction unit 1102E described below. Specifically, the control device 110 may interpolate the shape of the work object at the unobservable location by estimating the current shape of the work object based on the most recently observed shape of the work object and the trajectory (locus) of the work part when the work machine 100 most recently performed a predetermined operation.

[0171] For example, the trajectory prediction unit 1102D uses the learned model LM1 to predict the trajectory of the work part in the immediate future as a result of the operator's operation, based on data relating to the operation state of the work machine 100 and data representing the shape of the work object. The trajectory prediction unit 1102D may also predict the trajectory of the work part in the immediate future as a result of the operator's operation, using MPC (Model Predictive Control), based on data representing the operation state of the work machine 100 and data representing the shape of the work object. In this case, the machine learning unit 2005A may be omitted.

[0172] Incidentally, when the work machine 100 is a continuous unloader ULD, the multiple buckets 77 continue to move on the predetermined trajectory of the chain bucket 79 regardless of operator operation. Therefore, the trajectory prediction unit 1102D predicts the movement of the chain bucket on the predetermined trajectory. The work object shape prediction unit 1102E predicts the shape of the work object after a change due to a predetermined operation of the work machine 100 in the immediate future, based on the prediction result (predicted trajectory) of the trajectory prediction unit 1102D and the current shape of the work object. Specifically, the work object shape prediction unit 1102E acquires data representing the predicted shape of the work object after a change due to a predetermined operation of the work machine 100 in the immediate future, based on data representing the predicted trajectory and data representing the current shape of the work object, both acquired by the trajectory prediction unit 1102D. Furthermore, the work object shape prediction unit 1102E may predict the shape of the work object and its surrounding objects after changes due to a predetermined operation of the work machine 100 in the near future, based on the prediction results of the trajectory prediction unit 1102D and the shapes of the current work object and its surrounding objects. Specifically, the work object shape prediction unit 1102E may acquire data representing the predicted shapes of the work object and its surrounding objects after changes due to a predetermined operation of the work machine 100 in the near future, based on data representing the predicted trajectory and data representing the shapes of the current work object and its surrounding objects. This is because the predetermined operation of the work machine 100 on the work object may also affect the objects around the work object. The shape of the objects around the work object is, for example, the shape of the sediment surrounding the work area that continues from the sediment in the work area of ​​the excavator SVL. The data representing the shape of the objects around the work object is acquired, for example, based on output data from the sensor group 300 received via the communication device 180. Furthermore, data representing the shape of objects around the work target may be acquired based on output data from the periphery monitoring sensor 140 instead of or in addition to output data from the sensor group 300 .

[0173] For example, the work object shape prediction unit 1102E uses the learned model LM2 based on data representing the predicted trajectory and data representing the current shape of the work object to predict the shape of the work object after changes in the near future due to a predetermined operation of the work machine 100. Similarly, the work object shape prediction unit 1102E may use the learned model LM2 based on data representing the predicted trajectory and data representing the shapes of the current work object and surrounding objects to predict the shapes of the work object and surrounding objects after changes in the near future.

[0174] The safety level calculation unit 1102F evaluates the degree of safety (hereinafter "safety level") based on changes in the shape of the work object or the objects in its vicinity within a predetermined range (hereinafter "evaluation range" for convenience) corresponding to the work object or the work object and its surrounding objects. Specifically, the safety level calculation unit 1102F predicts changes in the shape of the work object, etc. within the evaluation range based on the prediction results of the work object shape prediction unit 1102E. Then, based on the prediction results of the change in the shape of the work object, etc., the safety level calculation unit 1102F calculates the safety level of a position within the evaluation range where a monitored object is detected by the object detection unit 1102B. Furthermore, the safety level calculation unit 1102F may calculate the safety level of a position where a monitored object may be present, instead of or in addition to the position where the monitored object is detected by the object detection unit 1102B. A position where a monitored object may exist is, for example, a position where occlusion occurs in the data representing the latest work object despite being within the observation range of the perimeter monitoring sensor 140 or the sensor 300-X. Hereinafter, the position where a monitored object is detected by the object detection unit 1102B or the position where a monitored object may exist will be referred to simply as the "position of the monitored object" for the sake of simplicity.

[0175] For example, the safety level calculation unit 1102F calculates the safety level of a position where the monitored object is detected so that the safety level decreases as the amount of change in the shape of the work target or surrounding objects at that position increases. This is because if the shape of the work target or surrounding objects changes, the monitored object may be caught in the change in shape. For example, as shown in Fig. 5, the safety level calculation unit 1102F calculates the safety level at a position where the monitored object is detected so that, with the amount of change being zero as a reference, the safety level decreases as the amount of change increases, and the rate of decrease in safety level also increases.

[0176] The safety level calculation unit 1102F may also calculate the safety level of a position based on the amount of change in the shape of the work target and surrounding objects within a relatively narrow predetermined range based on the position of the monitored object. This is because the safety of the monitored object may be reduced even if the shape of the work target or objects around the position of the monitored object changes. Furthermore, the monitored object may move. For example, the safety level calculation unit 1102F may calculate the safety level using the average value of the amount of change in the shape of the work target and surrounding objects within the predetermined range (hereinafter referred to as the "average change amount"). The safety level calculation unit 1102F may also calculate the safety level using the maximum value of the amount of change in the shape of the work target and surrounding objects within the predetermined range (hereinafter referred to as the "maximum change amount"). Specifically, the safety level calculation unit 1102F calculates the safety level at the position where the monitored object is detected such that the safety level decreases as the average change amount or maximum change amount within the predetermined range increases. For example, as in the case of Figure 5, the safety level calculation unit 1102F calculates the safety level of the position of the monitored object so that, with the average change amount and the maximum change amount being zero as a reference, the safety level decreases and the rate of decrease in safety level increases as the average change amount and the maximum change amount increase.

[0177] The safety level calculation unit 1102F can calculate the safety level from the amount of change in the shape of the work object, etc., using, for example, a conversion formula, conversion map, conversion table, etc., registered in advance in the auxiliary storage device 110A.

[0178] The safety control unit 1102G performs control relating to functional safety based on the safety level calculated by the safety level calculation unit 1102F.

[0179] The safety control unit 1102G activates the safety function, for example, when the safety level is relatively low compared to a predetermined standard. The safety level being relatively low compared to the predetermined standard may mean that the safety level is equal to or lower than the predetermined standard.

[0180] The safety functions activated by the safety control unit 1102G may be similar to those of the safety control unit 1102C.

[0181] For example, the safety control unit 1102G activates the internal alarm function or the remote alarm function when the safety level is relatively low compared to a predetermined standard (hereinafter referred to as the "alarm standard") specified for the alarm function. This allows the control device 110 to notify the operator that the operation of the work machine 100 may change the shape of the work target or its surroundings, potentially reducing the safety level. Therefore, the operator can ensure the safety of the work target or its surroundings by ceasing operation of the work machine 100. The safety control unit 1102G may also activate the external alarm function in addition to the internal alarm function or the remote alarm function. This allows the control device 110 to alert workers, supervisors, and others around the work machine 100. In this case, the alarm standard for the internal alarm function or the remote alarm function and the alarm standard for the external alarm function may be the same or different.

[0182] Furthermore, when the safety level is relatively low compared to the notification standard, the safety control unit 1102G may change the notification mode (i.e., the way of notification) depending on the level of the safety level.

[0183] For example, similar to the safety control unit 1102C, the safety control unit 1102G may selectively use a caution-level alarm and an alert-level alarm. Specifically, the safety control unit 1102G may output a caution-level alarm when the safety level is relatively high within a range where the safety level is relatively low relative to the alert standard, and may output an alert-level alarm when the safety level is relatively low. The relatively high and low safety levels are distinguished by a predetermined standard (hereinafter referred to as the "alert standard") that is lower than the normal alert standard (hereinafter referred to as the "alert standard" for convenience) that is specified for alert-level alarms. In other words, the safety control unit 1102G may output a caution-level alarm when the safety level is relatively low relative to the alert standard but not relatively low relative to the alert standard, and may output an alert-level alarm when the safety level is relatively low relative to the alert standard.

[0184] Furthermore, in a range where the safety level is relatively low compared to the notification standard, the safety control unit 1102G may change the notification mode in multiple stages of three or more or continuously depending on the level of the safety level in that range. Specifically, the safety control unit 1102G may issue a notification in a mode where the level of attention or vigilance increases as the safety level decreases in a range relatively low compared to the notification standard.

[0185] For example, if the operator stops the operation after the alarm function has started to operate, the safety control unit 1102G stops the alarm function.

[0186] Furthermore, the safety control section 1102G may activate the operation restriction function when the safety level is relatively low compared to a predetermined standard (hereinafter referred to as the "operation restriction standard") specified for the operation restriction function. This allows the control device 110 to restrict the operation of the work machine 100 when the operation of the work machine 100 may change the shape of the work object or its surroundings, thereby reducing the safety level. Therefore, the control device 110 can ensure the safety of the work object or its surroundings. The control device 110 may activate the operation deceleration function or the operation stop function when the safety level is relatively low compared to the operation restriction standard. Furthermore, an operation restriction standard for the operation deceleration function (hereinafter referred to as the "operation deceleration standard") and an operation restriction standard for the operation stop function (hereinafter referred to as the "operation stop standard") that is set lower than the operation deceleration standard may be provided. In this case, the safety control unit 1102G activates the operation deceleration function when the safety level is relatively low compared to the operation deceleration standard and not relatively low compared to the operation stop standard, and activates the operation stop function when the safety level is relatively low compared to the operation stop standard.

[0187] Furthermore, when the safety level is relatively low compared to the operation restriction standard, the safety control section 1103G may change the degree of operation restriction in three or more stages or continuously depending on the level of safety within that range. Specifically, the safety control section 1103G may activate the operation restriction function so that the lower the safety level is within a range where the safety level is relatively low compared to the operation restriction standard, the higher the level of operation restriction (i.e., the degree to which the operation of the work machine 100 is decelerated).

[0188] Furthermore, the safety control unit 1102G may activate the safety function taking into consideration the reliability of data representing the predicted shape of the work object, etc. (hereinafter referred to as "prediction reliability").

[0189] For example, the prediction reliability is set to be lower when the operator is not operating the operation device 130 than when the operator is operating it. This is because the accuracy of the data representing the predicted shape of the work object, etc. is higher when the actual operation by the operator is reflected. Similarly, the prediction reliability may be set to be lower when the work part is not moving than when it is moving. This is because the accuracy of the data representing the predicted shape of the work object is higher when the state in which the work part is actually moving is reflected. Furthermore, the prediction reliability may be set to be lower as the range of occlusion in the shape representing the current shape of the work object becomes larger. This is because when the range of occlusion in the shape representing the current shape of the work object becomes relatively larger, the accuracy of the predicted shape based on that shape becomes relatively lower.

[0190] For example, the safety control unit 1102G sets the predetermined standard for activating the safety function so that the higher the prediction reliability is, the higher the predetermined standard for activating the safety function is, and the lower the prediction reliability is, the lower the predetermined standard is set. This allows the control device 110 to make it more difficult to activate the safety function when the prediction reliability is relatively low. Therefore, the control device 110 can prevent a situation in which work efficiency is reduced due to inappropriate activation of the safety function when the prediction reliability is relatively low. Therefore, the control device 110 can achieve both safety for the work target and its surroundings and work efficiency.

[0191] Furthermore, when the safety level is relatively low compared to the notification standard, the safety control unit 1102G may change the notification mode depending on the level of prediction reliability.

[0192] For example, the safety control unit 1102G may change the pitch, sound pressure, tone, and / or sound cycle of the alarm sound output from the sound output device 164 and the remote operation support device 400 to increase the level of attention as the prediction reliability increases. The safety control unit 1102G may also change the color, shape, size, whether or not the alarm image flashes, and / or the flashing cycle of the alarm image on the display device 162 and the remote operation support device to increase the level of attention as the prediction reliability increases. The safety control unit 1102G may activate a visual notification function when the prediction reliability is relatively low, while activating an auditory notification function instead of or in addition to the visual notification function when the prediction reliability is relatively high. This allows the control device 110 to ensure safety of the work target and its surroundings while suppressing a decrease in work efficiency.

[0193] For example, if the operator stops operation after the operation restriction function has started, the safety control section 1102G stops the operation restriction function, which allows the operator to resume operation of the work machine 100.

[0194] In addition, the safety control unit 1102G may be capable of switching its function between ON (enabled) and OFF (disabled) in response to a predetermined input by an operator or the like to the input device 170 or the remote operation support device 400.

[0195] The safety control unit 1102G may activate the safety function only for monitored objects that are not targets for activation of the safety function of the safety control unit 1102C. In this case, the safety level calculation unit 1102F calculates the safety level of the positions of monitored objects that are detected by the object detection unit 1102B and that exist outside the activation range of the safety control unit 1102C (i.e., the above-mentioned notification range and operation restriction range).

[0196] [First Example of Processing of Operation Support System] Next, a first example of processing of the operation support system SYS will be described with reference to Fig. 6 and Fig. 7. Specifically, a specific example of processing will be described based on the first example of the operation support system SYS in Fig. 4.

[0197] Fig. 6 is a flowchart illustrating a first example of the processing of the operation support system SYS. Fig. 7 is a diagram illustrating an example of an observation target area.

[0198] This flowchart is executed repeatedly at predetermined processing intervals, for example, from the start to the end of a predetermined task performed by the work machine 100.

[0199] As shown in FIG. 6, in step S102, the control device 110 acquires a history of data relating to the operational state of the work machine 100, including data relating to the most recent operational state of the work machine 100 by the operator.

[0200] For example, the control device 110 may calculate the most recent trajectory (i.e., locus) x of the working portion of the work machine 100, which reflects the most recent history of the operation state of the work machine 100 by the operator. １：ｋ Obtain the orbit x １：ｋ is the state x of the work part for the most recent k processing cycles. ｎ (n=1, . . . , k), it is expressed by the following equation (1).

[0201]

[0202] State x of the work part at the nth time ｎ is m parameters x that define the position of the work part １ ｎ ~x ｍ ｎ Using (m≧2), it is expressed as the following equation (2).

[0203]

[0204] Parameter x １ ｎ ~x ｍ ｎare acquired based on output data of the operation monitoring sensor 150. For example, if the work machine 100 is an excavator SVL, the parameter defining the position of the working part is a parameter defining the position of the bucket 6. For example, the parameter defining the position of the bucket 6 is the attitude angle of the boom 4, arm 5, bucket 6, and upper rotating body 3 of the excavator SVL. Furthermore, if the work machine 100 is a continuous unloader ULD, which will be described later, the parameters defining the position of the working part are the attitude angles of the rotating body 55, boom 57, and scraping unit 61 of the continuous unloader ULD, as well as the center distance between the driven rollers 81b, 81c of the scraping unit 61.

[0205] When the process of step S102 is completed, the control device 110 proceeds to step S104.

[0206] In step S104, the control device 110 acquires data representing the latest shape of the work object, etc.

[0207] 7, an observation target area TA around the work machine 100, including the work target of the work machine 100, is divided into a predetermined number N of grids. The observation target area TA is the range of the area corresponding to the work target and its surrounding objects, whose shape is to be observed by the periphery monitoring sensor 140 or the periphery monitoring sensor 140 and the sensor group 300. The observation target area TA is also the area of ​​the area corresponding to the work target and its surrounding objects, whose shape is to be predicted by the work target shape prediction unit 1102E.

[0208] For example, the control device 110 acquires the latest shape h of the work object, etc. The shape h of the work object, etc. is calculated by multiplying the height h of each grid in the observation target area TA by the height h of each grid in the observation target area TA. ｉ (i=1, . . . , N), it is expressed as the following equation (3).

[0209]

[0210] Furthermore, the control device 110 may acquire an occlusion map o that indicates whether or not each lattice in the observation target area TA is occluded, in addition to the shape h of the work target, etc. The occlusion map o includes observability information o that indicates whether or not each lattice is occluded. ｊUsing (j=1, . . . , N), it is expressed as in the following equation (4).

[0211]

[0212] For example, the observation availability information o ｊ takes the value "1" if the state is observable, and takes the value "0" if the state is not observable.

[0213] In addition, the shape h of the work object and the occlusion map o are input, and the shape h of the work object is interpolated by the following formula (5) using a function q corresponding to the trained model. ｉｐ You may ask for:

[0214]

[0215] When the process of step S104 is completed, the control device 110 proceeds to step S106.

[0216] In step S106, the trajectory prediction unit 1102D of the control device 110 predicts the immediate future trajectory of the working portion of the work machine 100 based on the data acquired in steps S102 and S104, and acquires data representing the predicted trajectory.

[0217] For example, the trajectory prediction unit 1102D uses a function f corresponding to the learned model LM1 to obtain a predicted trajectory y of the work part using the following equation (6).

[0218]

[0219] Furthermore, a variable γ representing the characteristics of the soil or bulk cargo M to be worked on may be introduced, and the predicted trajectory y of the work area may be obtained using the following equation (7).

[0220]

[0221] The predicted trajectory y is the same as the trajectory x １：ｋ (see equation (1) above).

[0222] When the process of step S106 is completed, the control device 110 proceeds to step S108.

[0223] In step S108, work object shape prediction unit 1102E of control device 110 predicts the shape of the work object, etc., based on the data acquired in steps S104 and S106, and acquires data representing the predicted shape.

[0224] For example, the work object shape prediction unit 1102E uses a function r corresponding to the learned model LM2 to obtain a predicted shape h' of the work object, etc., using the following equation (8).

[0225]

[0226] The predicted shape h' of the work object, etc. is the predicted height h' of each grid in the observation target area TA, similar to the shape h of the work object, etc. ｓ Using (s=1, . . . , N), it is expressed as in the following equation (9).

[0227]

[0228] Furthermore, similar to the case of the above-mentioned equation (7), the work object shape prediction unit 1102E may introduce γ, which represents the characteristics of the work object, and obtain a predicted shape h′ of the work object, etc., using the following equation (10).

[0229]

[0230] Furthermore, the function r corresponding to the learned model LM2 uses the occlusion map o as input in addition to the current shape h of the work object, etc., and the predicted trajectory y, and calculates the predicted shape h of the work object, etc., using the following equation (11) or (12): ｉｐ ' may be obtained.

[0231]

[0232] When the process of step S108 is completed, the control device 110 proceeds to step S110.

[0233] In step S110, the control device 110 acquires data indicating the position of the monitored object. The position of the monitored object includes at least one of the position of the monitored object detected by the object detection unit 1102B as described above and a position where the monitored object may be present (for example, a position where the shape of a work object or the like cannot be observed due to occlusion).

[0234] When the process of step S110 is completed, the control device 110 proceeds to step S112.

[0235] In step S112, the safety level calculation unit 1102F of the control device 110 calculates the safety level for each position of the object to be monitored based on the data acquired in step S110.

[0236] For example, the safety level calculation unit 1102F calculates the amount of change in the shape of the work object, etc., based on the difference between the shape h of the work object, etc. and the predicted shape h'. Furthermore, when calculating the safety level of a position where the shape of the work object, etc. cannot be observed due to occlusion, the safety level calculation unit 1102F calculates the amount of change in the shape of the work object, etc., based on the difference between the shape h of the work object, etc. and the predicted shape h'. ｉｐ and predicted shape h ｉｐ The amount of change in the shape of the work target, etc. may be calculated from the difference between the calculated amount of change and the measured value of the safety level calculation unit 1102F. Then, the safety level calculation unit 1102F calculates the safety level for each position of the monitored object based on the calculated amount of change.

[0237] When the process of step S112 is completed, the control device 110 proceeds to step S114.

[0238] In step S114, the safety control unit 1102G of the control device 110 determines whether there is a location of a monitored object for which the calculation result (safety level) of step S112 is relatively low compared to a predetermined standard. If there is a location of a monitored object for which the safety level is relatively low compared to the predetermined standard, the safety control unit 1102G proceeds to step S116; otherwise, the processing of this flowchart is terminated.

[0239] When both the notification function and the operation restriction function are operable as safety functions, it is determined whether the safety level is relatively low for each of the notification standard and the operation restriction standard.

[0240] In step S116, the safety control section 1102G activates the safety function (that is, at least one of the alarm function and the operation restriction function).

[0241] When the process of step S116 is completed, the control device 110 ends the process of this flowchart.

[0242] [Second Example of Functional Configuration of Operation Support System] Next, a second example of the functional configuration of the operation support system SYS will be described with reference to FIG. 8 in addition to FIGS. 1 to 3.

[0243] In the following, the same symbols will be used for configurations that are the same as or correspond to those in the first example described above, and the explanation will focus on the differences from the first example described above, and explanations of the same or corresponding configurations as those in the first example described above may be omitted or simplified.

[0244] FIG. 8 is a functional block diagram showing a second example of the functional configuration of the operation support system SYS.

[0245] In this example, the operation support system SYS differs from the first example described above in that it provides support for the execution of work by a work machine 100 that operates using an autonomous driving function.

[0246] As shown in FIG. 8, the control device 110 of the work machine 100 includes, as functional units, an operation log providing unit 1101 and a work support unit 1102, similar to the first example described above.

[0247] As functional units, the information processing device 200 includes, similarly to the first example described above, a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a teacher data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007.

[0248] The teacher data generation unit 2004 includes teacher data generation units 2004B and 2004C.

[0249] The teacher data generation unit 2004C generates a teacher data set TRD3 for generating a trained model LM3.

[0250] The learned model LM3 uses data on the state of the work object of the work machine 100 (for example, the shape and characteristics of the work object) as input and infers the target trajectory of the work part for a specified operation of the work machine 100.

[0251] Each piece of training data included in the training data set TRD3 is a combination of the prior state of the work object as input data and the trajectory of the working part when the work machine 100 performs a predetermined operation as operated by an expert as correct output data. In other words, the training data generation unit 2004C generates the training data set based on the log acquired by the log acquisition unit 2001 of when the work machine 100 performs a predetermined operation as operated by an expert. Furthermore, if multiple types of predetermined operations are specified, a trained model LM3 may be generated for each type of predetermined operation. In this case, the training data generation unit 2004C generates the training data set TRD3 for each type of predetermined operation.

[0252] The machine learning unit 2005 includes machine learning units 2005B and 2005C.

[0253] The machine learning unit 2005C causes the base learning model M3 to perform machine learning based on the teacher data set output from the teacher data generation unit 2004C. In this way, the machine learning unit 2005C can generate a learned model LM3 that is capable of outputting (inferring) a target trajectory of a working part for a predetermined operation of the work machine 100, using data that represents the state of the work target of the work machine 100 as input.

[0254] The trained model storage unit 2006 stores the trained models LM2 and LM3 output by the machine learning unit 2005. Furthermore, when the machine learning unit 2005C re-learns or additionally trains the trained model LM3, the trained model LM3 in the trained model storage unit 2006 is updated.

[0255] The distribution unit 2007 distributes data of the learned models LM2 and LM3 to the work machine 100.

[0256] For example, when the machine learning unit 2005C generates or updates the learned model LM3, the distribution unit 2007 distributes the most recently generated or updated learned model LM3 to the work machine 100. In addition, the distribution unit 2007 may distribute the latest learned model LM3 in the learned model memory unit 2006 to the work machine 100 in response to a signal received from the work machine 100 requesting distribution of the learned model LM3.

[0257] Unlike the first example described above, the work support unit 1102 is a functional unit for providing work support to the work machine 100 operating using an autonomous driving function.

[0258] The work support unit 1102 includes a trained model storage unit 1102A, an object detection unit 1102B, a safety control unit 1102C, a work object shape prediction unit 1102E, a safety level calculation unit 1102F, and a safety control unit 1102G. Unlike the first example described above, the work support unit 1102 also includes a target trajectory generation unit 1102H and a motion control unit 1102I.

[0259] In addition, some or all of the functions of the object detection unit 1102B, safety control unit 1102C, work object shape prediction unit 1102E, safety level calculation unit 1102F, safety control unit 1102G, target trajectory generation unit 1102H, and operation control unit 1102I may be transferred to an outside of the work machine 100 (for example, to an information processing device 200).

[0260] The trained model storage unit 1102A stores trained models LM2 and LM3 that are distributed from the information processing device 200 and received via the communication device 180.

[0261] The target trajectory generation unit 1102H generates a target trajectory for the working part in a predetermined operation of the work machine 100 based on data representing the current state of the work object, and outputs the data representing the target trajectory of the working part. The data representing the current state of the work object is acquired, for example, from the periphery monitoring sensor 140. The data representing the current state of the work object may also be acquired from the sensor group 300 via the communication device 180.

[0262] For example, the target trajectory generating unit 1102H uses the learned model LM3 based on data representing the shape of the work object to generate a target trajectory for the working part during a predetermined operation of the work machine 100. Furthermore, the target trajectory generating unit 1102H may use the learned model LM3 to generate a target trajectory for the working part during a predetermined operation of the work machine 100 based on data representing the state of the work object in addition to the data representing the shape of the work object.

[0263] The target trajectory generating unit 1102H may apply any known method instead of the learned model LM3 to generate a target trajectory for the working part of the work machine 100 that is tailored to the state of the work object of the work machine 100. In this case, the teacher data generating unit 2004C and the machine learning unit 2005C may be omitted. For example, the target trajectory generating unit 1102H may use MPC (Model Predictive Control) to generate data that represents a target trajectory for the working part in a predetermined operation of the work machine 100, based on data that represents the current state of the work object. The target trajectory generating unit 1102H may also generate data that represents a target trajectory for the working part in a predetermined operation of the work machine 100 by optimizing a predetermined reference trajectory for the working part of the work machine 100, based on data that represents the state of the work object.

[0264] The operation control unit 1102I causes the work machine 100 to perform a predetermined operation so that a predetermined part of the work machine 100 moves along the target trajectory generated by the target trajectory generation unit 1102H. Specifically, the operation control unit 1102I controls the actuator 120 while determining the position of the working part from the output of the operation monitoring sensor 150, etc., to cause the work machine 100 to perform a predetermined operation so that the working part of the work machine 100 moves along the target trajectory. This allows the work machine 100 to autonomously proceed with work while executing a predetermined operation in accordance with the state of the work object, such as the shape and characteristics of the work object.

[0265] The work object shape prediction unit 1102E predicts the shape of the work object after a change due to a predetermined operation in the near future of the work machine 100, based on the target trajectory of the work part generated by the target trajectory generation unit 1102H and the current shape of the work object. Specifically, the work object shape prediction unit 1102E acquires data representing the predicted shape of the work object after a change required for a predetermined operation in the near future of the work machine 100, based on data representing the target trajectory of the work part, data representing the current shape of the work object, and n. The work object shape prediction unit 1102E may also predict the shape of the work object and its surrounding objects after a change due to a predetermined operation in the near future of the work machine 100, based on the target trajectory of the work part generated by the target trajectory generation unit 1102H and the current shapes of the work object and its surrounding objects. Specifically, the work object shape prediction unit 1102E may obtain data representing the predicted shape of the work object and its surrounding objects after changes due to a specified operation of the work machine 100 in the near future, based on data representing the target trajectory of the work area and data representing the shape of the current work object and its surrounding objects.

[0266] For example, the work object shape prediction unit 1102E uses the learned model LM2 based on data representing the target trajectory of the work part and data representing the current shape of the work object to predict the shape of the work object after it has changed in the near future due to a predetermined operation of the work machine 100. Similarly, the work object shape prediction unit 1102E may use the learned model LM2 based on data representing the target trajectory of the work part and data representing the shape of the current work object and surrounding objects to predict the shape of the work object and surrounding objects after it has changed in the near future.

[0267] The safety control unit 1102G performs control relating to functional safety based on the safety level calculated by the safety level calculation unit 1102F, as in the first example described above.

[0268] For example, the safety control unit 1102G activates the remote notification function when the safety level is relatively low compared to the notification standard. This allows the control device 110 to notify, for example, the remote monitoring supervisor that the operation of the work machine 100 may change the shape of the work object or its surroundings, potentially reducing the safety level. Therefore, the supervisor can ensure the safety of the work object or its surroundings by intervening in the autonomous driving function of the work machine 100 and slowing or stopping the operation of the work machine 100. Furthermore, the safety control unit 1102G may activate an external notification function in addition to the remote notification function.

[0269] The safety control unit 1102G stops the alarm function, for example, if the autonomous driving function of the work machine 100 is deactivated by the intervention of a remote monitoring supervisor after the alarm function has started to operate, or if the work machine 100 is stopped after the deactivation.

[0270] Furthermore, the safety control section 1102G may activate the operation restriction function when the safety level is relatively low compared to the operation restriction standard. This allows the control device 110 to restrict the operation of the work machine 100 in response to an operation command from the autonomous driving function when there is a possibility that the shape of the work object or its surroundings will change due to operation by the autonomous driving function of the work machine 100, thereby reducing the safety level. As a result, the control device 110 can ensure the safety of the work object or its surroundings.

[0271] For example, if the autonomous driving function is deactivated by an intervention operation of the remote monitoring supervisor after the operation restriction function has started, the safety control unit 1102G stops the operation restriction function. This allows the remote monitoring supervisor to, for example, confirm the safety conditions of the work target of the work machine 100 and its surroundings, and then resume the specified work using the autonomous driving function of the work machine 100. Furthermore, the remote monitoring supervisor can, for example, confirm the safety conditions of the work target of the work machine 100 and its surroundings, and then resume the specified work of the work machine 100 by operating it himself.

[0272] [Second Example of Processing of Operation Support System] Next, a second example of processing of the operation support system SYS will be described with reference to Fig. 9. Specifically, a specific example of processing based on the second example of the operation support system SYS in Fig. 8 will be described.

[0273] The following description will focus on the differences from the first example (FIG. 6) described above, and descriptions of the same or corresponding parts as the first example described above may be omitted or simplified.

[0274] As shown in FIG. 9, the process of step S202 is the same as the process of step S104 in FIG. 6, and therefore a description thereof will be omitted.

[0275] When the process of step S202 is completed, the control device 110 proceeds to step S204.

[0276] In step S204, the target trajectory generating unit 1102H of the control device 110 generates a target trajectory for the working part of the work machine 100, and acquires data representing the target trajectory for the working part.

[0277] For example, the data representing the target trajectory of the working part is the trajectory x １：ｋ and the predicted trajectory y.

[0278] When the process of step S204 is completed, the control device 110 proceeds to step S206.

[0279] In step S206, work object shape prediction unit 1102E of control device 110 predicts the shape of the work object, etc., based on the data acquired in steps S202 and S204, and acquires data representing the predicted shape.

[0280] When the process of step S206 is completed, the control device 110 proceeds to step S208.

[0281] The processes in steps S208, S210, S212, and S214 are the same as the processes in steps S110, S112, S114, and S116 in FIG. 6, and therefore will not be described further.

[0282] [Example of Application of Operation Support System to Excavator] Next, an example of application of the operation support system SYS to an excavator SVL will be described with reference to FIGS. 10 to 15. FIG.

[0283] <Configuration of Excavator> FIGS. 10 and 11 are a side view and a top view of an example of an excavator SVL to which the operation support system SYS is applied.

[0284] 10 and 11 , an excavator SVL as a work machine 100 includes a lower traveling body 1, an upper rotating body 3, an attachment AT including a boom 4, an arm 5, and a bucket 6, and a cabin 10. Hereinafter, the front of the excavator SVL corresponds to the direction in which the attachment extends from the upper rotating body 3 when the excavator SVL is viewed in a plan view from directly above along the rotation axis of the upper rotating body 3 (hereinafter simply referred to as a "plan view"). Furthermore, the left and right sides of the excavator SVL correspond to the left and right sides, respectively, as viewed from the operator inside the cabin 10.

[0285] The lower traveling body 1 includes, for example, a pair of left and right crawlers 1C. Specifically, the pair of left and right crawlers 1C includes a left crawler 1CL and a right crawler 1CR. The lower traveling body 1 causes the excavator SVL to travel by hydraulically driving each of the crawlers 1CL, 1CR by a traveling hydraulic motor 1M. Specifically, the traveling hydraulic motor 1M includes a left traveling hydraulic motor 1ML that drives the crawler 1CL, and a right traveling hydraulic motor 1MR that drives the crawler 1CR.

[0286] The upper rotating body 3 is mounted on the lower traveling body 1 so as to be rotatable via a rotating mechanism 2. The upper rotating body 3 rotates relative to the lower traveling body 1, for example, when the rotating mechanism 2 is hydraulically driven by a rotating hydraulic motor 2A.

[0287] The attachment AT corresponds to the work device 125 .

[0288] The boom 4 is attached to the front center of the upper rotating body 3 so as to be able to tilt up and down, and an arm 5 is attached to the tip of the boom 4 so as to be able to rotate up and down, and a bucket 6 is attached to the tip of the arm 5 so as to be able to rotate up and down.

[0289] The bucket 6 is an example of an end attachment and corresponds to a working part of the excavator SVL. The bucket 6 is used, for example, for excavation work. Furthermore, instead of the bucket 6, another end attachment may be attached to the tip of the arm 5 depending on the type of work, etc. The other end attachment may be, for example, another type of bucket, such as a slope bucket or a dredging bucket. The other end attachment may also be a type of end attachment other than a bucket, such as a mixer or a breaker.

[0290] The boom 4, arm 5, and bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, which serve as hydraulic actuators, respectively.

[0291] The cabin 10 is a cab in which an operator sits, and is mounted, for example, on the front left side of the upper rotating body 3.

[0292] The excavator SVL includes various components, such as a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside world, and a control system for various controls.

[0293] <Hydraulic Drive System> As shown in Figures 10 and 11 , the hydraulic drive system of the excavator SVL includes hydraulic actuators serving as actuators 120 that drive driven elements. The driven elements of the excavator SVL include, for example, crawlers 1CL, 1CR of the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6. The hydraulic actuators serving as actuators 120 include, for example, traveling hydraulic motors 1ML, 1MR, a swing hydraulic motor 2A, a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9. The hydraulic drive system of the excavator SVL also includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0294] The engine 11 is the main power source of the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, on the rear of the upper rotating body 3. The engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by, for example, a control device 110, and drives the main pump 14 and the pilot pump 15.

[0295] The regulator 13 adjusts the discharge amount of the main pump 14 under the control of the control device 110. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as the "tilt angle") in response to a control command from the control device 110.

[0296] The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the control device 110, the tilt angle of the swash plate is adjusted by the regulator 13, thereby adjusting the stroke length of the piston and controlling the discharge flow rate and discharge pressure.

[0297] The control valve 17 controls the hydraulic actuators in accordance with the details of the operator's operation of the operating device 130 or remote operation, or operation commands related to the automatic operation function. The operation commands related to the automatic operation function may be output from the control device 110 or from another control device. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. The control valve 17 selectively supplies hydraulic oil supplied from the main pump 14 to multiple hydraulic actuators in accordance with the details of the operation of the operating device 130 or operation commands corresponding to the automatic operation function.

[0298] <Operation System> As shown in FIGS. 10 and 11 , the operation system of the excavator SVL according to this embodiment includes a pilot pump 15 and an operation device 130.

[0299] The pilot pump 15 supplies pilot pressure to various hydraulic devices via a pilot line. The pilot pump 15 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

[0300] The pilot pump 15 may be omitted. In this case, for example, the pressure of the hydraulic oil supplied from the main pump 14 is reduced, and pilot pressure is supplied to various hydraulic devices through a pilot line.

[0301] The operation device 130 is provided near the driver's seat in the cabin 10 and is used by the operator to operate the various driven elements. In other words, the operation device 130 is used by the operator to operate the hydraulic actuators that drive the respective driven elements. The operation device 130 includes, for example, lever devices that operate the boom 4 (boom cylinder 7), the arm 5 (arm cylinder 8), the bucket 6 (bucket cylinder 9), and the upper rotating body 3 (swing hydraulic motor 2A). The operation device 130 also includes, for example, pedal devices and lever devices that operate the left and right crawlers 1CL, 1CR (travel hydraulic motors 1ML, 1MR) of the lower traveling body 1.

[0302] <User Interface System> As shown in FIGS. 10 and 11 , the user interface system of the excavator SVL includes an operation device 130 , an output device 160 , and an input device 170 .

[0303] The output device 160 includes a display device 162 and a sound output device 164 as described above.

[0304] <Communication System> As shown in FIGS. 10 and 11 , the communication system of the excavator SVL according to this embodiment includes a communication device 180.

[0305] <Control System> As shown in FIGS. 10 and 11 , the control system of the excavator SVL includes a control device 110 , a periphery monitoring sensor 140 , and an operation monitoring sensor 150 .

[0306] The control device 110 performs various controls related to the excavator SVL.

[0307] The control device 110 controls the operation of the hydraulic actuator (driven element) of the excavator SVL, for example, with the above-mentioned operation control valve as the control object.

[0308] For example, the control device 110 may control the operation of a hydraulic actuator (driven element) of the excavator SVL based on the operation of the operation device 130, with the operation control valve being the control object.

[0309] Furthermore, the control device 110 may perform control relating to the remote operation of the hydraulic actuator (driven element) of the excavator SVL, with the operation control valve as the control target.

[0310] The control device 110 may also control the automatic operation function of the excavator SVL, with the operation control valve as the control target.

[0311] Furthermore, the control device 110 controls the operation support system SYS as described above.

[0312] The periphery monitoring sensor 140 acquires and outputs data indicating the state of the periphery of the excavator SVL. The periphery monitoring sensor 140 includes an imaging device 141 and a distance measurement sensor 142.

[0313] It should be noted that either the imaging device 141 or the distance measurement sensor 142 may be omitted.

[0314] The imaging device 141 acquires an image showing the state of the periphery of the shovel SVL. The imaging device 141 is, for example, a monocular camera. The imaging device 141 may also be a 3D camera that can acquire data showing distance (depth) in addition to two-dimensional images. When the imaging device 141 is a 3D camera, the distance measurement sensor 142 may be omitted. The imaging device 141 includes cameras 141B, 141F, 141L, and 141R.

[0315] Cameras 141B, 141F, 141L, and 141R are attached to the upper rear end, upper front end, upper left end, and upper right end of the upper rotating body 3, respectively, and capture images of the rear, front, left side, and right side of the upper rotating body 3. For example, cameras 141B, 141L, and 141R are attached to the upper surface of the house part of the upper rotating body 3. Also, for example, camera 141F is attached to the upper surface of the cabin 10 of the upper rotating body 3. Also, cameras 141B, 141F, 141L, and 141R are each attached to the upper part of the upper rotating body 3 with their optical axes facing diagonally downward, and capture images in the vertical imaging range that includes from the ground near the excavator SVL to areas far from the excavator SVL. Furthermore, the cameras 141B, 141F, 141L, and 141R may be configured so that their vertical orientation, that is, their rotation angles, can be changed in response to an operation by a user such as an operator.

[0316] The cameras 141B, 141F, 141L, and 141R output captured images at predetermined intervals (e.g., 1 / 30 seconds) from the time the excavator SVL is started (i.e., the key switch is turned ON) until it is stopped (i.e., the key switch is turned OFF). The captured images output from the cameras 141B, 141F, 141L, and 141R are taken into the control device 110.

[0317] Note that some of the cameras 141B, 141F, 141L, and 141R may be omitted. For example, when it is assumed that an operator is on board the cabin 10, at least one of the cameras 141F and 141L may be omitted. This is because the operator can relatively easily grasp the situation in front of and to the left of the upper rotating body 3 from the cabin 10 by visually checking through the rearview mirror or window.

[0318] The ranging sensor 142 acquires data indicating the distance to an object in the vicinity of the shovel SVL. The ranging sensor 142 is, for example, a LIDAR. The ranging sensor 142 may also be a millimeter-wave radar, an ultrasonic sensor, an infrared sensor, or the like. The following description will focus on the case where the ranging sensor 142 is a LIDAR. The ranging sensor 142 includes ranging sensors 142BL, 142BR, 142L, and 142R.

[0319] Distance measurement sensors 142BL, 142BR, 142L, and 142R are attached to the upper left rear end, the upper right rear end, the upper left end, and the upper right end of the upper rotating body 3, respectively. Distance measurement sensors 142BL, 142BR, 142L, and 142R acquire data indicating the distance to objects to the left rear, right rear, left side, and right side of the upper rotating body 3, respectively. Distance measurement sensors 142BL, 142BR, 142L, and 142R are attached to the upper part of the upper rotating body 3 so that the reference axis of the infrared radiation direction faces diagonally downward, and may have an infrared radiation range in the up-down direction centered on a part of the ground relatively close to the excavator SVL.

[0320] Distance measuring sensors 142BL, 142BR, 142L, and 142R may be, for example, a scanning LIDAR, which is a three-dimensional laser scanner capable of scanning the infrared laser radiation direction in the vertical and horizontal directions. Distance measuring sensors 142BL, 142BR, 142L, and 142R may also be, for example, a so-called flash LIDAR, which radiates infrared light from a light-emitting module over a wide three-dimensional area and captures the reflected light (infrared light) with a three-dimensional distance image element.

[0321] The distance measuring sensors 142BL, 142BR, 142L, and 142R each output data and reflections indicating the distance to an object in the vicinity of the shovel SVL at predetermined intervals from the start to the stop of the shovel SVL. The data output from the distance measuring sensors 142BL, 142BR, 142L, and 142R is taken into the control device 110.

[0322] The operation monitoring sensor 150 acquires data indicating the operating state of the shovel SVL. The operation monitoring sensor 150 includes sensors 151 to 155.

[0323] The sensor 151 is attached to the boom 4 and measures the attitude of the boom 4. The sensor 151 outputs measurement data representing the attitude of the boom 4. The attitude of the boom 4 is, for example, the attitude angle around the rotation axis of the base end of the boom 4, which corresponds to the connection part between the boom 4 and the upper rotating body 3. The sensor 151 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), etc. The same may apply to the sensors 152 to 154 below. The sensor 151 may also include a cylinder sensor that detects the extension / retraction position of the boom cylinder 7. The same may apply to the sensors 152 and 153 below. The output of the sensor 151 (measurement data representing the attitude of the boom 4) is input to the control device 110. This allows the control device 110 to grasp the attitude of the boom 4.

[0324] The sensor 152 is attached to the arm 5 and measures the posture of the arm 5. The sensor 152 outputs measurement data that indicates the posture of the arm 5. The posture of the arm 5 is, for example, the posture angle around the rotation axis of the base end of the arm 5, which corresponds to the connection part between the arm 5 and the boom 4. The output of the sensor 152 (measurement data that indicates the posture of the arm 5) is taken into the control device 110. This allows the control device 110 to grasp the posture of the arm 5.

[0325] The sensor 153 is attached to the bucket 6 and measures the attitude of the bucket 6. The sensor 153 outputs measurement data that indicates the attitude of the bucket 6. The attitude of the bucket 6 is, for example, the attitude angle around the rotation axis of the base end of the bucket 6 that corresponds to the connection part with the arm 5. The output of the sensor 153 (measurement data that indicates the attitude of the bucket 6) is taken into the control device 110. This allows the control device 110 to grasp the attitude of the bucket 6.

[0326] The sensor 154 measures the attitude state of the machine body of the shovel SVL (for example, the upper rotating body 3). The sensor 154 outputs measurement data representing the attitude state of the machine body of the shovel SVL. The attitude state of the machine body of the shovel SVL is, for example, the inclination state of the machine body with respect to a predetermined reference plane (for example, a horizontal plane). For example, the sensor 154 is attached to the upper rotating body 3 and measures the inclination angle of the shovel SVL about two axes in the forward / backward direction and the left / right direction. The output of the sensor 154 (measurement data representing the attitude state of the machine body of the shovel SVL) is taken into the control device 110. This allows the control device 110 to grasp the attitude state (inclination state) of the machine body (upper rotating body 3).

[0327] The sensor 155 is attached to the upper rotating body 3 and measures the rotation state of the upper rotating body 3. The sensor 155 outputs measurement data representing the rotation state of the upper rotating body 3. The sensor 155 measures, for example, the rotation angular velocity and rotation angle of the upper rotating body 3. The sensor 155 includes, for example, a gyro sensor, a resolver, a rotary encoder, etc. The output of the sensor 155 (measurement data representing the rotation state of the upper rotating body 3) is input to the control device 110. This allows the control device 110 to grasp the rotation state of the upper rotating body 3, such as the rotation angle.

[0328] The outputs of the sensors 151 to 155 are input to the control device 110. This allows the control device 110 to estimate the position of the bucket 6, which serves as the working part at the tip of the attachment AT, based on the outputs of the sensors 151 to 155.

[0329] If the sensor 154 includes a gyro sensor, a six-axis sensor, an IMU, or the like that can detect angular velocities around three axes, the rotation state (e.g., rotation angular velocity) of the upper rotating body 3 may be detected based on the detection signal of the sensor 154. In this case, the sensor 155 may be omitted.

[0330] <Specific Examples of Operation of Operation Support System> FIGS. 12 to 15 are diagrams showing first to fourth examples of changes in the shape of earth and sand around the shovel due to the digging operation of the shovel.

[0331] First Example As shown in FIG. 12, in this example, a shovel SVL and a worker W1 around the shovel SVL are working in an area corresponding to a work target of the shovel SVL.

[0332] A worker W1 is positioned in front of the excavator SVL, and the object detection unit 1102B of the control device 110 can detect the worker W1 as an object to be monitored based on the image captured by the camera 141F. However, because the position of the worker W1 is some distance away from the upper rotating body 3 of the excavator SVL, the safety control unit 1102C of the control device 110 does not activate the alarm function or the operation restriction function, and the excavator SVL can continue working.

[0333] Here, the excavator SVL is moving the bucket 6 along the track F120 in response to an operator operation or by using an automatic driving function, and is attempting to excavate the soil and sand that is the work target.

[0334] When the excavator SVL performs an excavation operation along the track F120, the soil around the track F120 is removed, which may cause the soil around the track F120 to flow into the location of the track F120. Therefore, in this example, the current shape F121 of the soil to be worked on changes to shape F122, which results in a significant change in the shape of the soil where the worker W1 is located, which may cause the worker W1 to be swept away by the soil (see the black arrow in the figure).

[0335] In contrast, in this example, the control device 110 predicts the shape F122 of the work target from the trajectory F120 of the bucket 6 and calculates the safety level of the position where the worker W1 is located based on the change from the shape F121 to the shape F122. Since the change in the shape of the soil and sand that is the work target at the position where the worker W1 is located is relatively large, the safety level becomes relatively low compared to the predetermined standard, and the safety control unit 1102G of the control device 110 can activate safety functions such as an alarm function and an operation restriction function. Therefore, the control device 110 can prompt the operator to stop operation in response to activation of the internal alarm function or the remote alarm function, or can slow down or stop the operation of the excavator SVL in response to activation of the operation restriction function. Therefore, the control device 110 can suppress the change from the shape F121 to the shape F122 and ensure the safety of the worker W1.

[0336] Second Example As shown in FIG. 13, in this example, a shovel SVL is working in a work area at the foot of a slope, and a worker W2 is working on the upper surface of the slope on an extension of the peak.

[0337] In this example, the worker W2 is located in a blind spot of the perimeter monitoring sensor 140 of the shovel SVL. Therefore, the object detection unit 1102B of the control device 110 cannot detect the worker W2 even if it uses the output of the perimeter monitoring sensor 140. On the other hand, the object detection unit 1102B of the control device 110 can detect the worker W2 based on the output of the sensor 300-X acquired via the communication device 180. However, because the position of the worker W2 is far away from the shovel SVL, the safety control unit 1102C of the control device 110 does not activate the alarm function or the operation restriction function, and the shovel SVL can continue working.

[0338] Here, the excavator SVL is about to excavate earth and sand at the foot of the slope in response to the operation of the operator or by using an automatic operation function.

[0339] When the excavator SVL excavates soil at the foot of the slope, the soil above the excavation point may collapse and flow downward. Therefore, in this example, the shape F131 of the work target and the soil around it changes to shape F132, which significantly changes the shape of the soil where worker W2 is located, potentially causing worker W2 to be swept away by the soil (see the black arrow in the figure).

[0340] In contrast, in this example, the control device 110 predicts the shape F132 of the work target after the excavation operation of the shovel SVL changes, and calculates the safety level for the location where the worker W2 is located based on the change from shape F131 to shape F132. Because the change in the shape of the soil and sand that is the work target at the location where the worker W2 is located is relatively large, the safety level becomes relatively low compared to the predetermined standard, and the safety control unit 1102G of the control device 110 can activate safety functions such as an alarm function and an operation restriction function. Therefore, the control device 110 can prompt the operator to stop the operation in response to activation of the internal alarm function or remote alarm function, or can slow down or stop the operation of the shovel SVL in response to activation of the operation restriction function. Therefore, the control device 110 can suppress the change from shape F131 to shape F132 and ensure the safety of the worker W2.

[0341] <<Third Example>> As shown in Figure 14, in this example, the excavator SVL is working in a working area on the upper surface of the slope, which is an extension of the top of the slope, and the worker W4 is working on the lower surface of the slope, which is an extension of the base of the slope.

[0342] In this example, worker W4 is located in a blind spot of the perimeter monitoring sensor 140 of the shovel SVL. Therefore, the object detection unit 1102B of the control device 110 cannot detect worker W4 even if it uses the output of the perimeter monitoring sensor 140. On the other hand, the object detection unit 1102B of the control device 110 can detect worker W4 based on the output of sensor 300-X acquired via the communication device 180. However, because the position of worker W2 is far away from the shovel SVL, the safety control unit 1102C of the control device 110 does not activate the alarm function or the operation restriction function, and the shovel SVL can continue working.

[0343] Here, the excavator SVL is about to excavate earth and sand from the top of the slope in response to the operation of the operator or by using an automatic operation function.

[0344] When the excavator SVL excavates the soil at the top of the slope, the soil near the top of the slope may collapse and flow down the slope (see the black arrow in the figure). Therefore, in this example, the shape F141 of the work object and the soil around it may change to shape F142, and as a result, the soil that has flowed out may reach the location where the worker W3 is located.

[0345] In contrast, in this example, the control device 110 predicts the shape F142 of the work target after the excavation operation of the shovel SVL changes, and calculates the safety level for the location of the worker W3 based on the change from shape F141 to shape F142. Because the change in the shape of the soil and sand that is the work target at the location where the worker W3 is located is relatively large, the safety level becomes relatively low compared to the predetermined standard, and the safety control unit 1102G of the control device 110 can activate safety functions such as an alarm function and an operation restriction function. Therefore, the control device 110 can prompt the operator to stop the operation in response to activation of the internal alarm function or remote alarm function, or can slow down or stop the operation of the shovel SVL in response to activation of the operation restriction function. Therefore, the control device 110 can suppress the change from shape F141 to shape F142 and ensure the safety of the worker W3.

[0346] <Fourth Example> As shown in FIG. 15 , in this example, a shovel SVL is working in one work area separated by an obstacle OB, the lower part of which is buried underground, and a worker W4 is working in the other work area.

[0347] In this example, worker W4, who is the object to be monitored, is located in the blind spot of the perimeter monitoring sensor 140 of the shovel SVL due to the influence of obstacle OB. Therefore, the object detection unit 1102B of the control device 110 cannot detect worker W4 even using the output of the perimeter monitoring sensor 140. Therefore, the safety control unit 1102C of the control device 110 does not activate the alarm function or the operation restriction function, and the shovel SVL can continue working.

[0348] Furthermore, the object detection unit 1102B of the control device 110 can detect the worker W4 based on the output of the sensor 300-X when it is possible to acquire the output of the sensor 300-X at the work site via the communication device 180. However, because the position of the worker W4 is far away from the shovel SVL, the safety control unit 1102C of the control device 110 does not activate the alarm function or the operation restriction function, and the shovel SVL can continue working.

[0349] Furthermore, in this example, the control device 110 can detect an obstacle OB as a monitored object based on an image captured by the camera 141F included in the perimeter monitoring sensor 140. However, because the position of the worker W4 is some distance away from the upper rotating body 3 of the excavator SVL, the safety control unit 1102C of the control device 110 does not activate the alarm function or the operation restriction function, and the excavator SVL can continue working.

[0350] Here, the excavator SVL has begun excavating earth and sand near the base of the obstacle OB in response to the operator's operation or by using an automatic driving function.

[0351] When the excavator SVL excavates soil near the base of the obstacle OB, the soil at the base of the obstacle OB may flow into the path of the excavation operation bucket 6. Therefore, in this example, the current shape F151 of the soil to be worked on changes to shape F152, which may cause the fixed state of the obstacle OB to become unstable and cause the obstacle OB to tip over toward the worker W4 (see the black arrow in the figure).

[0352] In contrast, in this example, the control device 110 predicts the shape F152 of the work object after the excavation operation of the shovel SVL changes, and calculates the safety level of the location where the obstacle OB is located based on the change from shape F151 to shape F152. Since the change in the shape of the soil and sand in the work object at the location where the obstacle OB is located is relatively large, the safety level becomes relatively low compared to the predetermined standard, and the safety control unit 1102G of the control device 110 can activate safety functions such as an alarm function and an operation restriction function. Therefore, the control device 110 can prompt the operator to stop operation in response to activation of the internal alarm function or remote alarm function, or can slow down or stop the operation of the shovel SVL in response to activation of the operation restriction function. Therefore, the control device 110 can suppress the change from shape F151 to shape F152 and ensure the safety of the worker W4.

[0353] [Example of Application of Operation Support System to Continuous Unloader] Next, an example of application of the operation support system SYS to a continuous unloader ULD will be described with reference to FIGS. 16 to 20. FIG.

[0354] <Configuration of Continuous Unloader> Figures 16 to 19 are diagrams showing an example of an application of the operation support system SYS to a continuous unloader ULD. Specifically, Figure 16 is an overall view showing an example of a continuous unloader ULD. Figures 17 to 19 are diagrams showing a front view, a side view, and a cross-sectional view of a bucket elevator 59, respectively.

[0355] 16, the continuous unloader ULD (an example of a work machine) is a so-called bucket elevator type that is installed at a quay QY and continuously carries out (unloads) onto land bulk cargo M in the hold HD of a ship SP that is berthed at the quay QY. The bulk cargo M is, for example, coal, coke, iron ore, etc.

[0356] The quay QY is constructed of, for example, reinforced concrete, and two rails 53 are installed on the quay QY in the direction of its extension, i.e., parallel to the longitudinal direction of the ship SP to be docked. The continuous unloader ULD is configured to be movable on the two rails 53 and stops at a predetermined position while unloading from the ship SP.

[0357] The continuous unloader ULD includes a traveling section 52, a revolving body 55, a boom 57, a bucket elevator 59, and a cab 66. The component group including the boom 57 and the bucket elevator 59 corresponds to the above-described working device 125.

[0358] The running part 52 is configured to be movable on two rails 53 of the quay QY.

[0359] The rotating body 55 is mounted on the running part 52 so as to be able to rotate.

[0360] The boom 57 is provided to extend forward from the rotating body 55 from the quay QY to the part of the sea where the vessel SP is to come alongside, and is configured to be able to be raised and lowered relative to the rotating body 55. Specifically, the boom 57 can be raised and lowered in accordance with the extension and contraction of a cylinder 65 attached between the boom 57 and the rotating body 55.

[0361] The bucket elevator 59 is provided at the tip of the boom 57 so as to extend downward, i.e., toward the ship SP (hold HD). As shown in Figures 16 to 19, the bucket elevator 59 has a scraping unit 61 at its tip, and bulk goods M scraped by the scraping unit 61 are transported upward by a bucket 77 and landed on land. The scraping unit 61 corresponds to a working position.

[0362] 18, the scraping unit 61 can change its angle of inclination in the fore-and-aft direction relative to the elevator body 64. For example, when scraping loose cargo M directly below the opening OP of the hold HD, the scraping unit 61 is used in a state where it is not inclined in the fore-and-aft direction relative to the elevator body 64 (the state shown by the solid line in the figure). On the other hand, when scraping loose cargo M located closer to the inner wall than the opening OP of the hold HD, the scraping unit 61 is used in a state where it is inclined forward relative to the elevator body 64 (the state shown by the dashed dotted line in the figure).

[0363] A parallel link 58 is provided between the rotating body 55 and the bucket elevator 59, and the action of the parallel link 58 allows the bucket elevator 59 to maintain a vertical state regardless of the hoisting angle of the boom 57. The bucket elevator 59 can also move up and down in response to the up and down hoisting of the boom 57. A counterweight 63 is supported on the rotating body 55 via a link extending rearward on the opposite side from the boom 57, and a balancing lever 62 is provided to connect the counterweight 63 and the boom 57. This allows the load to be balanced between the bucket elevator 59 and the counterweight 63.

[0364] The operator's cab 66 is provided at the front of the rotating body 55 (i.e., at the position in the direction in which the boom 57 extends), and is used by an operator to sit in and operate the continuous unloader ULD.

[0365] The operator's cab 66 is provided with, for example, an operating device 130. This allows an operator to use the operating device 130 to operate driven elements such as the rotating body 55, the boom 57, and the bucket elevator 59.

[0366] The cab 66 is also equipped with, for example, a control device 110. The cab 66 is also equipped with, for example, an output device 160 and an input device 170.

[0367] When the continuous unloader ULD is used to unload bulk cargo M, an operator is stationed in the cab 66, and a worker is stationed inside the hold HD to notify the operator of the status of the bulk cargo. The operator in the cab 66 and the worker in the hold HD then work together to unload the bulk cargo M using the continuous unloader ULD.

[0368] The operator in the driver's cab 66 can operate the continuous unloader ULD while checking the image information displayed on the display device 162, which shows the opening OP of the cargo hold HD and the condition inside it, as well as the contact information from the worker about the condition inside the cargo hold HD.

[0369] As shown in Figures 17 and 18, the bucket elevator 59 includes an elevator body 64 that extends in the vertical direction, and a chain bucket 79 that moves in a circular motion between an upper portion 59a and a lower portion (scraping portion 61) of the bucket elevator 59.

[0370] The chain bucket 79 includes a pair of roller chains 75 and a plurality of buckets 77 supported so as to be suspended from the pair of roller chains 75 .

[0371] The roller chain 75 passes through the inside of the elevator body 64 and is connected endlessly between the upper part 59 a and the lower part (scraping part 61 ) of the bucket elevator 59 .

[0372] The bucket elevator 59 also includes a drive roller 81 a around which the roller chain 75 is stretched, driven rollers 81 b and 81 c that guide the roller chain 75 , and a diverting roller 83 .

[0373] The drive roller 81a is provided on the upper portion 59a of the bucket elevator 59, and the driven rollers 81b and 81c are provided in the scraping unit 61 at a predetermined interval in the front-rear direction.

[0374] The diverting roller 83 is disposed below the drive roller 81 a in the upper portion 59 a of the bucket elevator 59 and is configured to be able to change the traveling direction of the roller chain 75 .

[0375] A cylinder 85 is connected between the driven rollers 81b, 81c, and the distance between the axes of the driven rollers 81b, 81c is changed according to the extension and contraction of the cylinder 85, thereby changing the circular movement locus of the chain bucket 79.

[0376] The roller chain 75 is driven by a drive roller 81a and moves in a circular motion in the direction of arrow W relative to the elevator body 64. The chain bucket 79 circulates between the upper portion 59a of the bucket elevator 59 and the scraping portion 61 while moving in a circular motion.

[0377] In the scraping section 61, as the roller chain 75 moves in a substantially horizontal direction from driven roller 81b toward 81c, the bucket 77 scrapes loose goods M from the opening of the bucket 77 into its interior. The bucket 77, which has scraped and stored loose goods M, rises with its opening facing upward as the roller chain 75 rises from driven roller 81c toward drive roller 81a. When the bucket 77 arrives at the upper part 59a of the bucket elevator 59, its opening turns downward as the roller chain 75 changes direction from upward to downward as it passes through drive roller 81a. As a result, the loose goods M inside the bucket 77 are sent from a discharge chute to a rotary feeder 87 provided on the outer periphery of the bucket elevator 59.

[0378] The rotary feeder 87 transports the bulk goods M sent from the bucket 77 through a discharge chute to a boom conveyor 89 provided on the boom 57 .

[0379] The boom conveyor 89 is disposed inside the boom 57. Bulk goods M from the rotary feeder 87 of the bucket elevator 59 are transferred onto the boom conveyor 89 and transported toward the rotating body 55. A hopper is provided at the end of the boom conveyor 89 on the rotating body 55 side, and the bulk goods M transported by the boom conveyor 89 are supplied to the belt conveyor 93 through the hopper.

[0380] The belt conveyor 93 is provided on the running section 52. The belt conveyor 93 transports the bulk goods M to the ground belt conveyor 95. As a result, the bulk goods M are transported to the ground equipment 99 via the ground belt conveyor 95.

[0381] As shown in FIGS. 16 to 19, the continuous unloader ULD also includes a perimeter monitoring sensor 140 and an operation monitoring sensor 150 .

[0382] The periphery monitoring sensor 140 acquires and outputs data showing the state of the interior of the hold HD into which the bulk cargo M to be worked on is loaded. The periphery monitoring sensor 140 includes an imaging device 143 and an imaging device 144.

[0383] The imaging device 143 is attached to the upper part 59a of the bucket elevator 59 and is arranged so as to be able to capture images of the interior of the hold HD of the vessel SP from above through an opening OP at the top of the hold HD. The imaging device 143 includes cameras 143A to 143D. The cameras 143A to 143D are provided at equal intervals in the circumferential direction in a plan view on the outer circumferential surface of the upper part 59a of the bucket elevator 59.

[0384] The imaging device 144 is attached to the scraping unit 61 at the bottom of the bucket elevator 59, and is positioned so that it can capture images of the bulk goods M inside the hold HD from a relatively close distance. The imaging device 144 includes cameras 144A and 144B. The cameras 144A and 144B are provided on the left and right outer surfaces of the fixed part of the scraping unit 61. The cameras 144A and 144B are positioned so that their optical axes are directed downward and they can capture images of the bulk goods M from above.

[0385] The cameras 143A to 143D and the cameras 144A and 144B output captured images (image data) at predetermined intervals (e.g., 1 / 30 seconds) from the start to the stop of the continuous unloader ULD. The captured images output from the cameras 143A to 143D and the cameras 144A and 144B are taken into the control device 110.

[0386] Furthermore, the continuous unloader ULD (bucket elevator 59) may be provided with other types of perimeter monitoring sensors 140 (for example, distance measurement sensors) in addition to the imaging devices 143, 144. For example, one or more distance measurement sensors may be provided as the perimeter monitoring sensor 140 capable of acquiring point cloud data of a range including the imaging range of the imaging device 143. Furthermore, one or more distance measurement sensors may be provided as the perimeter monitoring sensor 140 capable of acquiring point cloud data of a range including the imaging range of the imaging device 144.

[0387] The operation monitoring sensors 150 acquire data representing the operating state of the continuous unloader ULD. The operation monitoring sensors 150 include sensors 156 to 159.

[0388] The sensor 156 is attached to the rotating unit 55 and measures the rotation state of the rotating unit 55. The sensor 156 outputs data indicating the rotation state of the rotating unit 55. The sensor 156 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a six-axis sensor, etc. The same may be true for the sensors 157 to 159 below.

[0389] The sensor 157 is attached to the boom 57 and measures the attitude of the boom 57. The sensor 157 outputs data representing the attitude of the boom 57.

[0390] The sensor 158 is attached to a fixed portion of the scraping unit 61 and measures the posture of the scraping unit 61. The sensor 158 outputs data representing the posture of the scraping unit 61.

[0391] The sensor 159 outputs data indicating the center distance between the driven rollers 81b and 81c in the scraping unit 61. The sensor 158 is, for example, a cylinder sensor that measures the expansion and contraction state of the cylinder 85.

[0392] The outputs of the sensors 156 to 159 are input to the control device 110. As a result, the control device 110 can estimate the position of the tip of the scraping unit 61 as the working part, i.e., the position of the part facing the bulk goods M, based on the outputs of the sensors 156 to 159.

[0393] <Specific Example of Operation of Operation Support System> FIG. 20 is a diagram showing an example of a change in the shape of the bulk goods M around the scraping section 61 due to the scraping operation of the continuous unloader ULD.

[0394] As shown in Figure 20, in this example, the scraping section 61 of the continuous unloader ULD is performing the scraping operation of loose cargo M inside the hold HD, and nearby there is a worker W5 who is checking the status of the loose cargo M and sending instructions to the operator via wireless communication, etc.

[0395] In this example, the object detection unit 1102B of the control device 110 can detect the worker W5 as the object to be monitored, based on the image captured by the imaging device 143. However, because the position of the worker W5 is some distance away from the scraping unit 61, the safety control unit 1102C of the control device 110 does not activate the alarm function or the operation restriction function, and the continuous unloader ULD can continue the unloading operation.

[0396] Here, when the scraping unit 61 performs a scraping operation, there is a possibility that the surrounding loose goods M will flow into the area scraped by the scraping unit 61. Therefore, in this example, the shape F201 of the current loose goods M to be worked on changes to shape F202, and as a result, the shape of the loose goods M at the location where the worker W5 is located changes significantly, which may cause the worker W5 to be swept away by the loose goods M (see the black arrow in the figure).

[0397] In contrast, in this example, the control device 110 predicts the shape F202 of the work object after the scraping operation of the scraping unit 61 and calculates the safety level for the location of the worker W5 based on the change from shape F201 to shape F202. Because the change in shape of the loose load M, the work object at the location where the worker W5 is located, is relatively large, the safety level becomes relatively low compared to the predetermined standard. The safety control unit 1102G of the control device 110 can activate safety functions such as an alarm function and a motion restriction function. Therefore, the control device 110 can prompt the operator to stop the operation in response to activation of the internal alarm function or remote alarm function, or can slow down or stop the scraping operation of the scraping unit 61 in response to activation of the motion restriction function. Therefore, the control device 110 can suppress the change from shape F201 to shape F202 and ensure the safety of the worker W5.

[0398] [Operation] Next, the operation of the work machine, information processing device, and program according to this embodiment will be described.

[0399] In a first aspect of this embodiment, the work machine is equipped with a control device. The work machine is, for example, the excavator SVL or continuous unloader ULD described above as work machine 100. The control device is, for example, the control device 110 described above. Specifically, the control device acquires information representing the shape of a work object, or information representing the shapes of the work object and objects around the work object, and predicts, based on that information, changes in the shape of the work object or the work object and objects around the work object that will occur in response to the operation of the work machine, and, based on the results of the predicted changes, issues a warning to an operator or to the outside, or restricts the operation of the work machine.

[0400] In a first aspect of this embodiment, the information processing device acquires information representing the shape of a work target of a work machine, or information representing the shapes of the work target and objects around the work target, and based on that information, predicts changes in the shape of the work target, or the work target and objects around the work target, in response to the operation of the work machine, and may issue a warning to an operator or to an outside of the work machine, or may restrict the operation of the work machine, based on the predicted change. The information processing device is, for example, the control device 110 or information processing device 200 described above.

[0401] Furthermore, in a first aspect of this embodiment, the program causes the information processing device to acquire information representing the shape of a work target of the work machine, or information representing the shapes of the work target and objects in the vicinity of the work target, and based on that information, predicts changes in the shape of the work target, or the work target and objects in the vicinity of the work target, in accordance with the operation of the work machine, and may issue a warning to an operator or outside the work machine, or may restrict the operation of the work machine, based on the predicted change. For example, the program is a program stored in the auxiliary storage device 110A of the control device 110 or the auxiliary storage device 202 of the information processing device 200.

[0402] This allows the work machine or information processing device (hereinafter referred to as "work machine, etc.") to predict changes in the shape of the work object, etc. in accordance with the operation of the work machine, and, for example, in situations where a decrease in safety is predicted due to the change in shape, to issue an alert to the operator and urge them to stop operation. Similarly, the work machine, etc. can predict changes in the work object, etc. in accordance with the operation of the work machine, and, for example, in situations where a decrease in safety is predicted due to the change in shape, to issue an alert to the outside and urge monitored objects, such as people, to evacuate from areas where safety is decreased. Similarly, the work machine, etc. can predict changes in the work object, etc. in accordance with the operation of the work machine, and, for example, in situations where a decrease in safety is predicted due to the change in shape, to restrict the operation of the work machine and suppress changes in the shape of the work object, etc. As a result, the work machine, etc. can further improve safety around the work machine.

[0403] Furthermore, in a second aspect of this embodiment, based on the first aspect described above, the control device or the information processing device (hereinafter referred to as "the control device, etc.") may evaluate the safety of a location based on the predicted change result at a location where a monitored object within the range of the predicted change has been detected, or at a location where the monitored object may be present, and may issue the alert or restrict the operation of the work machine based on the evaluation result.

[0404] This allows work machines, etc. to ensure the safety of monitored objects in situations where the safety of the monitored objects may be reduced due to changes in the shape of the work target, etc. in response to the operation of the work machine.

[0405] Furthermore, in a third aspect of this embodiment, based on the second aspect described above, the control device etc. may evaluate the safety of a location based on the predicted result of the change within a predetermined range including the location where the monitored object is detected within the range of the predicted change, or the location where the monitored object may be present.

[0406] This allows work machines, etc. to ensure the safety of the monitored object, for example, in situations where there is a possibility that the shape of the work object, etc. will change relatively significantly within a specified range including the position of the monitored object.

[0407] Furthermore, in a fourth aspect of this embodiment, assuming any one of the first to third aspects described above, the control device etc. may predict the operation of the work machine based on information representing the shape of the work object, and may predict the change based on the predicted result of the operation of the work machine and the information representing the shape of the work object.

[0408] This allows the work machine or the like to predict the operation of the work machine from the current shape of the work object, and to predict changes in the shape of the work object from the prediction results.

[0409] Furthermore, in a fifth aspect of this embodiment, assuming any one of the first to third aspects described above, the control device may acquire information representing the operating state of the work machine by the operator, predict the operation of the work machine based on that information and information representing the shape of the work object, and predict the change based on the predicted result of the operation of the work machine and the information representing the shape of the work object.

[0410] This allows the work machine or the like to predict the operation of the work machine from the shape of the work object and the operating state of the work machine, and to predict changes in the shape of the work object from the prediction results.

[0411] In a sixth aspect of this embodiment, based on any one of the first to third aspects described above, the work machine may have an automatic driving function. The control device or the like may acquire information related to a motion plan for the work machine using the automatic driving function, and predict the change based on that information and information representing the shape of the work object, or information representing the shapes of the work object and objects in its vicinity. The information related to the motion plan for the work machine is, for example, data representing the target trajectory of the work portion described above.

[0412] This allows the work machine, etc. to predict changes in the shape of the work object, etc. from the operation plan of the work machine using the automatic driving function.

[0413] In a seventh aspect of the present embodiment, based on any one of the first to sixth aspects described above, the work machine may be provided with a sensor that outputs information representing the shape of the work object, or the shape of the work object and its surroundings. The sensor is, for example, the periphery monitoring sensor 140 described above.

[0414] This allows the work machine or the like to acquire information representing the shape of the work object or the like using sensors mounted on the work machine.

[0415] In an eighth aspect of this embodiment, based on any one of the first to seventh aspects described above, the work machine or the like may be equipped with a communication device for communicating with the outside. The communication device is, for example, the communication device 180 described above. The control device or the like may then acquire information representing the shape of at least one of the work target and its surrounding objects from outside the work machine via the communication device. The information representing the shape of at least one of the work target and its surrounding objects acquired from the outside is, for example, output data from the sensor group 300 described above.

[0416] This allows the work machine etc. to use sensors etc. external to the work machine to acquire information that represents the shape of the work target etc. Therefore, the work machine etc. can predict changes in the shape of the work target etc. over a wider range, for example.

[0417] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist described in the claims.

[0418] Finally, this application claims priority based on Japanese Patent Application No. 2023-204120, filed December 1, 2023, the entire contents of which are incorporated herein by reference.

[0419] REFERENCE SIGNS LIST 1 Lower travelling body 3 Upper rotating body 4 Boom 5 Arm 6 Bucket 52 Travelling section 55 Rotating body 57 Boom 59 Bucket elevator 61 Scraping section 100 Work machine 110 Control device 120 Actuator 125 Work device 130 Operation device 140 Periphery monitoring sensor 141 Imaging device 141B, 141F, 141L, 141R Camera 142 Distance measuring sensor 142BL, 142BR, 142L, 142R Distance measuring sensor 143 Imaging device 143A to 143D Camera 144 Imaging device 144A, 144B Camera 150 Operation monitoring sensor 151 to 155 Sensors 156 to 159 Sensors 160 Output device 162 Display device 164 LIST OF REFERENCE SIGNS 170 Sound output device 170 Input device 180 Communication device 200 Information processing device 300 Sensor group 300-1 to 300-M Sensor 400 Remote operation support device 1101 Operation log providing unit 1101A Operation log recording unit 1101B Operation log storage unit 1101C Operation log transmission unit 1102 Work support unit 1102A Model storage unit 1102B Object detection unit 1102C Safety control unit 1102D Trajectory prediction unit 1102E Work object shape prediction unit 1102F Safety degree calculation unit 1102G Safety control unit 1102H Target trajectory generation unit 1102I Operation control unit 1103G Safety control unit 2001 Log acquisition unit 2002 Simulator unit 2003 Log storage unit 2004 Teacher data generation unit 2004A Teacher data generation unit 2004B Teacher data generation unit 2004C Teacher data generation unit 2005 Machine learning unit 2005A Machine learning unit 2005B Machine learning unit 2005C Machine learning unit 2006 Model storage unit 2007 Distribution unit HD Hold M Bulk cargo SLV Shovel SP Ship SVL Shovel SYS Operation support system ULD Continuous unloader

Claims

1. A work machine equipped with a control device that acquires information representing the shape of a work object, or information representing the shapes of said work object and objects surrounding said work object, and based on that information, predicts changes in the shape of said work object, or said work object and objects surrounding said work object, in response to the operation of the work machine, and issues a warning to an operator or to the outside, or limits the operation of the work machine, based on the results of the predicted changes.

2. A work machine as described in claim 1, wherein the control device evaluates the safety of a position where a monitored object within the predicted change range has been detected or where the monitored object may be present based on the predicted change result, and issues the alert or restricts the operation of the work machine based on the evaluation result.

3. A work machine as described in claim 2, wherein the control device evaluates the safety of a position based on the predicted change within a predetermined range including a position where the monitored object is detected within a range of the predicted change, or a position where the monitored object may be present.

4. A work machine as claimed in any one of claims 1 to 3, wherein the control device predicts the operation of the work machine based on information representing the shape of the work object, and predicts the change based on the predicted result of the work machine operation and the information representing the shape of the work object.

5. A work machine as claimed in any one of claims 1 to 3, wherein the control device acquires information representing the operating state of the work machine by an operator, predicts the operation of the work machine based on that information and information representing the shape of the work object, and predicts the change based on the predicted result of the work machine operation and the information representing the shape of the work object.

6. A work machine having an automatic driving function, comprising: a control device for acquiring information relating to an operation plan of the work machine using the automatic driving function, and predicting the change based on the information and information representing the shape of the work object, or information representing the shape of the work object and objects surrounding it. A work machine as described in any one of claims 1 to 3.

7. A work machine as claimed in any one of claims 1 to 3, comprising a sensor that outputs information representing the shape of the work target, or the shapes of the work target and surrounding objects.

8. A work machine as claimed in any one of claims 1 to 3, comprising a communication device for communicating with the outside, wherein the control device acquires information representing the shape of at least one of the work target and its surrounding objects from the outside via the communication device.

9. An information processing device which acquires information representing the shape of a work target of a work machine, or information representing the shape of the work target and objects surrounding the work target, predicts changes in the shape of the work target or the work target and objects surrounding the work target in response to the operation of the work machine based on that information, and issues an alert to an operator or outside the work machine based on the predicted changes, or restricts the operation of the work machine.

10. A program which causes an information processing device to obtain information representing the shape of a work target of a work machine, or information representing the shapes of the work target and objects surrounding the work target, predict, based on that information, changes in the shape of the work target, or the work target and objects surrounding the work target, in response to the operation of the work machine, and, based on the predicted changes, issue an alert to an operator or outside the work machine, or restrict the operation of the work machine.

Images